MyFavMirrors/moonfire-nvr
1
0
Fork 0
mirror of https://github.com/scottlamb/moonfire-nvr.git synced 2025-01-15 08:45:01 -05:00
Code Issues Packages Projects Releases Wiki Activity
moonfire-nvr/server/src/web/mod.rs

1633 lines
60 KiB
Rust
Raw Normal View History

upgrade copyright notices * As discussed in #48, say "The Moonfire NVR Authors" at the top of every file rather than whoever created that file. Have one AUTHORS file listing everyone. * Consistently call it a "security camera network video recorder" rather than "security camera digital video recorder".
2020-03-01 22:53:41 -08:00
// This file is part of Moonfire NVR, a security camera network video recorder.
start splitting up web.rs It's getting huge and hard to work with. The path stuff is easy to pull out.
2021-10-28 12:38:29 -07:00
// Copyright (C) 2021 The Moonfire NVR Authors; see AUTHORS and LICENSE.txt.
shorten per-file copyright headers I'm tired of all the boilerplate, so use the new GPL-3.0-linking-exception license identifier instead in all the server components. I left the ui stuff alone because I'm just going to replace it (#111). Add a checker for the header because it's easy to forget.
2021-02-17 13:28:48 -08:00
// SPDX-License-Identifier: GPL-v3.0-or-later WITH GPL-3.0-linking-exception.
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 14:34:00 -08:00
start splitting up web.rs It's getting huge and hard to work with. The path stuff is easy to pull out.
2021-10-28 12:38:29 -07:00
mod path;
split static file serving into its own file
2021-10-28 12:49:50 -07:00
mod static_file;
start splitting up web.rs It's getting huge and hard to work with. The path stuff is easy to pull out.
2021-10-28 12:38:29 -07:00
use self::path::Path;
more readable async web code This uses "async fn" throughout rather than a mix of async and the older futures style. And it takes advantage of the "self: Arc<Self>" syntax to avoid having a ServiceInner. It was confusing to have some methods on Service and some on ServiceInner; now that distinction is gone. One downside is there's a little more atomic reference-counting. Before, service_fn essentially took an &Arc<Self>, which means it could call Arc::clone where its use of self actually outlived the future (see stream_live_m4s) but didn't need to otherwise. After, it calls an async fn that takes Arc<Self>. Using &Arc<Self> is apparently possible (as of Rust 1.41) but using that with "async fn" means the returned future is tied to its lifetime. The workaround is to use async blocks as described here: <https://rust-lang.github.io/async-book/03_async_await/01_chapter.html> but that's really ugly: it brings back the explicit Future reference, requires futures::future::Either in some cases, and introduces another level of indenting. I think it's better to just pay the arc costs which are probably negligible, or at least cheaper than the boxing was before. Oh, and I make this compile on Rust 1.40 again as it claimed to. http-serve accidentally used the &Arc<Self> thing which broke this. Update to a freshly-pushed commit which doesn't do that.
2020-05-30 19:08:54 -07:00
use crate::body::Body;
lose "extern crate" everywhere (Rust 2018 edition)
2018-12-28 21:53:29 -06:00
use crate::json;
use crate::mp4;
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
use base::{bail_t, ErrorKind};
small readability improvements to web.rs I think this has a minor behavior change: permission denied replies change to HTTP 403 where they were HTTP 401. The new behavior seems more correct, as these errors can occur when authentication has succeeded but the session in question is not authorized for the given operation. The UI currently doesn't care about this distinction.
2021-03-10 16:18:47 -08:00
use base::{clock::Clocks, format_err_t};
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 14:34:00 -08:00
use core::borrow::Borrow;
use core::str::FromStr;
lose "extern crate" everywhere (Rust 2018 edition)
2018-12-28 21:53:29 -06:00
use db::dir::SampleFileDir;
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
use db::{auth, recording};
use typed errors in /view.mp4 path This fixes #178. Before, everything got translated to 5xx status; now it produces the correct type in several cases. Ideally I'd get rid of the untyped errors in all of web.rs; this is a small step.
2021-10-21 10:25:37 -07:00
use failure::{format_err, Error};
support multiple sample file directories This is still pretty basic support. There's no config UI support for renaming/moving the sample file directories after they are created, and no error checking that the files are still in the expected place. I can imagine sysadmins getting into trouble trying to change things. I hope to address at least some of that in a follow-up change to introduce a versioning/locking scheme that ensures databases and sample file dirs match in some way. A bonus change that kinda got pulled along for the ride: a dialog pops up in the config UI while a stream is being tested. The experience was pretty bad before; there was no indication the button worked at all until it was done, sometimes many seconds later.
2018-02-11 22:45:51 -08:00
use fnv::FnvHashMap;
switch to websocket for live stream (#59) The multipart stream / hanging GET approach worked in a prototype for a single stream, but Chrome has a per-host limit of six connections. If I try streaming all my cameras at once, I hit that limit. I can't open all the streams, much less additional connections to load init segments and such. Websockets apparently has a much higher limit of 256.
2020-02-28 21:41:31 -08:00
use futures::stream::StreamExt;
send keepalives on live.m4s Chrome appears to time out at 60 seconds of inactivity otherwise. I think it's better to keep the stream open, even if the camera is broken. The implementation looks awkward, but that might be the state of Rust async right now.
2021-03-25 23:01:38 -07:00
use futures::{future::Either, sink::SinkExt};
upgrade to hyper 0.12.x Just one (intentional) functional change---now the streamers start shutting down while the webserver shuts down gracefully.
2018-08-29 22:26:19 -07:00
use http::header::{self, HeaderValue};
UI preferences: #153 #155
2021-09-01 15:01:42 -07:00
use http::method::Method;
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
use http::{status::StatusCode, Request, Response};
overhaul HTTP serving and caching * use content-hashed paths for static resources (except the top-level request), with immutable Cache-Control headers. This should improve cache behavior in both directions: avoid preventable HTTP requests and cause immediate refresh when needed. I had some staleness when browsing with my phone. * set up the favicons properly while I'm at it (closes #50). I used the convenient favicons-webpack-plugin to build everything from a .svg. I've hit an error similar to lovell/sharp#1593 at least once though so I might change my mind about that part if it continues to be problematic. * use http-serve's new directory traversal code for static file serving. This removes the odd behavior where files that weren't present at server startup couldn't be served. (I wasn't comfortable switching to the content-hashed paths before doing this.) It also means the static files can be served compressed. JSON API responses were already served compressed, so this closes #25. * for a given API URL, decide if we want it to be cached or not server-side. Stop using jQuery's kludgy cache-defeating _=<timestamp> URL parameter. I might start setting etags on some of these things and could serve 304 Not Modified responses if it's genuinely unmodified.
2020-05-29 21:20:14 -07:00
use http_serve::dir::FsDir;
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
use hyper::body::Bytes;
logging improvements * allow debug/trace logging on release builds again * enable log messages from hyper. I didn't notice they went away with 0.14.0, although there's a breaking change in the log: https://github.com/hyperium/hyper/blob/master/CHANGELOG.md#v0140-2020-12-23 * downgrade some particularly spammy messages
2021-06-04 23:28:48 -07:00
use log::{debug, info, trace, warn};
overhaul HTTP serving and caching * use content-hashed paths for static resources (except the top-level request), with immutable Cache-Control headers. This should improve cache behavior in both directions: avoid preventable HTTP requests and cause immediate refresh when needed. I had some staleness when browsing with my phone. * set up the favicons properly while I'm at it (closes #50). I used the convenient favicons-webpack-plugin to build everything from a .svg. I've hit an error similar to lovell/sharp#1593 at least once though so I might change my mind about that part if it continues to be problematic. * use http-serve's new directory traversal code for static file serving. This removes the odd behavior where files that weren't present at server startup couldn't be served. (I wasn't comfortable switching to the content-hashed paths before doing this.) It also means the static files can be served compressed. JSON API responses were already served compressed, so this closes #25. * for a given API URL, decide if we want it to be cached or not server-side. Stop using jQuery's kludgy cache-defeating _=<timestamp> URL parameter. I might start setting etags on some of these things and could serve 304 Not Modified responses if it's genuinely unmodified.
2020-05-29 21:20:14 -07:00
use memchr::memchr;
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
use nom::bytes::complete::{tag, take_while1};
replace regex use with nom This reduces the binary size noticeably on my macOS machine (#70): unstripped stripped 1 before switching to clap 11.1 MiB 6.7 MiB 2 after switching to clap 11.4 MiB 6.9 MiB 3 without regex 10.1 MiB 5.9 MiB
2020-04-17 23:02:02 -07:00
use nom::combinator::{all_consuming, map, map_res, opt};
use nom::sequence::{preceded, tuple};
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
use nom::IResult;
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-20 22:08:18 -08:00
use std::cmp;
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
use std::convert::TryFrom;
support proxy forwarded headers I went with legacy headers (X-Real-IP, X-Forwarded-Proto) because they appear to be more widely supported than the RFC 7239 Forwarded header.
2018-11-28 14:22:30 -08:00
use std::net::IpAddr;
fully implement json handling as in spec This is a significant milestone; now the Rust branch matches the C++ branch's features. In the process, I switched from using serde_derive (which requires nightly Rust) to serde_codegen (which does not). It was easier than I thought it'd be. I'm getting close to no longer requiring nightly Rust.
2016-12-08 21:28:50 -08:00
use std::ops::Range;
style improvements * remove stuttering: mp4::Mp4Foo -> mp4::Foo * stop using a &MutexGuard<Foo> where a &Foo will do
2017-02-24 21:33:26 -08:00
use std::sync::Arc;
switch to websocket for live stream (#59) The multipart stream / hanging GET approach worked in a prototype for a single stream, but Chrome has a per-host limit of six connections. If I try streaming all my cameras at once, I hit that limit. I can't open all the streams, much less additional connections to load init segments and such. Websockets apparently has a much higher limit of 256.
2020-02-28 21:41:31 -08:00
use tokio_tungstenite::tungstenite;
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 14:34:00 -08:00
use url::form_urlencoded;
use uuid::Uuid;
small readability improvements to web.rs I think this has a minor behavior change: permission denied replies change to HTTP 403 where they were HTTP 401. The new behavior seems more correct, as these errors can occur when authentication has succeeded but the session in question is not authorized for the given operation. The UI currently doesn't care about this distinction.
2021-03-10 16:18:47 -08:00
/// An HTTP error response.
/// This is a thin wrapper over the hyper response type; it doesn't even verify
/// that the response actually uses a non-2xx status code. Its purpose is to
/// allow automatic conversion from `base::Error`. Rust's orphan rule prevents
/// this crate from defining a direct conversion from `base::Error` to
/// `hyper::Response`.
struct HttpError(Response<Body>);
impl From<Response<Body>> for HttpError {
fn from(response: Response<Body>) -> Self {
HttpError(response)
}
}
impl From<base::Error> for HttpError {
fn from(err: base::Error) -> Self {
HttpError(from_base_error(err))
}
}
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
fn plain_response<B: Into<Body>>(status: http::StatusCode, body: B) -> Response<Body> {
Response::builder()
.status(status)
.header(header::CONTENT_TYPE, HeaderValue::from_static("text/plain"))
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
.body(body.into())
.expect("hardcoded head should be valid")
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
}
small readability improvements to web.rs I think this has a minor behavior change: permission denied replies change to HTTP 403 where they were HTTP 401. The new behavior seems more correct, as these errors can occur when authentication has succeeded but the session in question is not authorized for the given operation. The UI currently doesn't care about this distinction.
2021-03-10 16:18:47 -08:00
fn not_found<B: Into<Body>>(body: B) -> HttpError {
HttpError(plain_response(StatusCode::NOT_FOUND, body))
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
}
small readability improvements to web.rs I think this has a minor behavior change: permission denied replies change to HTTP 403 where they were HTTP 401. The new behavior seems more correct, as these errors can occur when authentication has succeeded but the session in question is not authorized for the given operation. The UI currently doesn't care about this distinction.
2021-03-10 16:18:47 -08:00
fn bad_req<B: Into<Body>>(body: B) -> HttpError {
HttpError(plain_response(StatusCode::BAD_REQUEST, body))
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
}
small readability improvements to web.rs I think this has a minor behavior change: permission denied replies change to HTTP 403 where they were HTTP 401. The new behavior seems more correct, as these errors can occur when authentication has succeeded but the session in question is not authorized for the given operation. The UI currently doesn't care about this distinction.
2021-03-10 16:18:47 -08:00
fn internal_server_err<E: Into<Error>>(err: E) -> HttpError {
HttpError(plain_response(
StatusCode::INTERNAL_SERVER_ERROR,
err.into().to_string(),
))
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
}
introduce typed errors and use in mp4 code Fixes #46. If there are no video_sample_entries, it returns InvalidArgument, which gets mapped to a HTTP 400. Various other failures turn into non-500s as well. There are many places that can & should be using typed errors, but it's a start.
2018-12-28 17:30:33 -06:00
fn from_base_error(err: base::Error) -> Response<Body> {
small readability improvements to web.rs I think this has a minor behavior change: permission denied replies change to HTTP 403 where they were HTTP 401. The new behavior seems more correct, as these errors can occur when authentication has succeeded but the session in question is not authorized for the given operation. The UI currently doesn't care about this distinction.
2021-03-10 16:18:47 -08:00
use ErrorKind::*;
introduce typed errors and use in mp4 code Fixes #46. If there are no video_sample_entries, it returns InvalidArgument, which gets mapped to a HTTP 400. Various other failures turn into non-500s as well. There are many places that can & should be using typed errors, but it's a start.
2018-12-28 17:30:33 -06:00
let status_code = match err.kind() {
small readability improvements to web.rs I think this has a minor behavior change: permission denied replies change to HTTP 403 where they were HTTP 401. The new behavior seems more correct, as these errors can occur when authentication has succeeded but the session in question is not authorized for the given operation. The UI currently doesn't care about this distinction.
2021-03-10 16:18:47 -08:00
Unauthenticated => StatusCode::UNAUTHORIZED,
PermissionDenied => StatusCode::FORBIDDEN,
InvalidArgument | FailedPrecondition => StatusCode::BAD_REQUEST,
NotFound => StatusCode::NOT_FOUND,
introduce typed errors and use in mp4 code Fixes #46. If there are no video_sample_entries, it returns InvalidArgument, which gets mapped to a HTTP 400. Various other failures turn into non-500s as well. There are many places that can & should be using typed errors, but it's a start.
2018-12-28 17:30:33 -06:00
_ => StatusCode::INTERNAL_SERVER_ERROR,
};
plain_response(status_code, err.to_string())
}
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-20 22:08:18 -08:00
#[derive(Debug, Eq, PartialEq)]
struct Segments {
ids: Range<i32>,
allow listing and viewing uncommitted recordings There may be considerable lag between being fully written and being committed when using the flush_if_sec feature. Additionally, this is a step toward listing and viewing recordings before they're fully written. That's a considerable delay: 60 to 120 seconds for the first recording of a run, 0 to 60 seconds for subsequent recordings. These recordings aren't yet included in the information returned by /api/?days=true. They probably should be, but small steps.
2018-03-02 11:38:11 -08:00
open_id: Option<u32>,
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-20 22:08:18 -08:00
start_time: i64,
end_time: Option<i64>,
}
update various deps This brings most things reasonably up-to-date. libpasta's deps are dragging a bit, keeping us on an older ring to avoid duplication, and causing us to use three versions of base64. And I need to update a few of my companion crates' parking_lot dep to match tokio.
2020-11-23 00:23:03 -08:00
fn num<'a, T: FromStr>() -> impl FnMut(&'a str) -> IResult<&'a str, T> {
replace regex use with nom This reduces the binary size noticeably on my macOS machine (#70): unstripped stripped 1 before switching to clap 11.1 MiB 6.7 MiB 2 after switching to clap 11.4 MiB 6.9 MiB 3 without regex 10.1 MiB 5.9 MiB
2020-04-17 23:02:02 -07:00
map_res(take_while1(|c: char| c.is_ascii_digit()), FromStr::from_str)
}
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-20 22:08:18 -08:00
impl Segments {
replace regex use with nom This reduces the binary size noticeably on my macOS machine (#70): unstripped stripped 1 before switching to clap 11.1 MiB 6.7 MiB 2 after switching to clap 11.4 MiB 6.9 MiB 3 without regex 10.1 MiB 5.9 MiB
2020-04-17 23:02:02 -07:00
/// Parses the `s` query parameter to `view.mp4` as described in `design/api.md`.
/// Doesn't do any validation.
fn parse(i: &str) -> IResult<&str, Segments> {
// Parse START_ID[-END_ID] into Range<i32>.
// Note that END_ID is inclusive, but Ranges are half-open.
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
let (i, ids) = map(
tuple((num::<i32>(), opt(preceded(tag("-"), num::<i32>())))),
|(start, end)| start..end.unwrap_or(start) + 1,
)(i)?;
replace regex use with nom This reduces the binary size noticeably on my macOS machine (#70): unstripped stripped 1 before switching to clap 11.1 MiB 6.7 MiB 2 after switching to clap 11.4 MiB 6.9 MiB 3 without regex 10.1 MiB 5.9 MiB
2020-04-17 23:02:02 -07:00
// Parse [@OPEN_ID] into Option<u32>.
let (i, open_id) = opt(preceded(tag("@"), num::<u32>()))(i)?;
// Parse [.[REL_START_TIME]-[REL_END_TIME]] into (i64, Option<i64>).
let (i, (start_time, end_time)) = map(
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
opt(preceded(
tag("."),
tuple((opt(num::<i64>()), tag("-"), opt(num::<i64>()))),
)),
|t| t.map(|(s, _, e)| (s.unwrap_or(0), e)).unwrap_or((0, None)),
)(i)?;
Ok((
i,
Segments {
ids,
open_id,
start_time,
end_time,
},
))
replace regex use with nom This reduces the binary size noticeably on my macOS machine (#70): unstripped stripped 1 before switching to clap 11.1 MiB 6.7 MiB 2 after switching to clap 11.4 MiB 6.9 MiB 3 without regex 10.1 MiB 5.9 MiB
2020-04-17 23:02:02 -07:00
}
}
impl FromStr for Segments {
type Err = ();
fn from_str(s: &str) -> Result<Self, Self::Err> {
let (_, s) = all_consuming(Segments::parse)(s).map_err(|_| ())?;
if s.ids.end <= s.ids.start {
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-20 22:08:18 -08:00
return Err(());
}
replace regex use with nom This reduces the binary size noticeably on my macOS machine (#70): unstripped stripped 1 before switching to clap 11.1 MiB 6.7 MiB 2 after switching to clap 11.4 MiB 6.9 MiB 3 without regex 10.1 MiB 5.9 MiB
2020-04-17 23:02:02 -07:00
if let Some(e) = s.end_time {
if e < s.start_time {
return Err(());
}
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-20 22:08:18 -08:00
}
replace regex use with nom This reduces the binary size noticeably on my macOS machine (#70): unstripped stripped 1 before switching to clap 11.1 MiB 6.7 MiB 2 after switching to clap 11.4 MiB 6.9 MiB 3 without regex 10.1 MiB 5.9 MiB
2020-04-17 23:02:02 -07:00
Ok(s)
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-20 22:08:18 -08:00
}
}
add concept of user/session permissions (I also considered the names "capabilities" and "scopes", but I think "permissions" is the most widely understood.) This is increasingly necessary as the web API becomes more capable. Among other things, it allows: * non-administrator users who can view but not access camera passwords or change any state * workers that update signal state based on cameras' built-in motion detection or a security system's events but don't need to view videos * control over what can be done without authenticating Currently session permissions are just copied from user permissions, but you can also imagine admin sessions vs not, as a checkbox when signing in. This would match the standard Unix workflow of using a non-administrative session most of the time. Relevant to my current signals work (#28) and to the addition of an administrative API (#35, including #66).
2019-06-19 15:17:50 -07:00
struct Caller {
permissions: db::Permissions,
UI preferences: #153 #155
2021-09-01 15:01:42 -07:00
user: Option<json::ToplevelUser>,
add concept of user/session permissions (I also considered the names "capabilities" and "scopes", but I think "permissions" is the most widely understood.) This is increasingly necessary as the web API becomes more capable. Among other things, it allows: * non-administrator users who can view but not access camera passwords or change any state * workers that update signal state based on cameras' built-in motion detection or a security system's events but don't need to view videos * control over what can be done without authenticating Currently session permissions are just copied from user permissions, but you can also imagine admin sessions vs not, as a checkbox when signing in. This would match the standard Unix workflow of using a non-administrative session most of the time. Relevant to my current signals work (#28) and to the addition of an administrative API (#35, including #66).
2019-06-19 15:17:50 -07:00
}
small readability improvements to web.rs I think this has a minor behavior change: permission denied replies change to HTTP 403 where they were HTTP 401. The new behavior seems more correct, as these errors can occur when authentication has succeeded but the session in question is not authorized for the given operation. The UI currently doesn't care about this distinction.
2021-03-10 16:18:47 -08:00
type ResponseResult = Result<Response<Body>, HttpError>;
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 14:34:00 -08:00
read-only signals support (#28) This is mostly untested and useless by itself, but it's a starting point. In particular: * there's no way to set up signals or add/remove/update events yet except by manual changes to the database. * if you associate a signal with a camera then remove the camera, hitting /api/ will error out.
2019-06-06 16:18:13 -07:00
fn serve_json<T: serde::ser::Serialize>(req: &Request<hyper::Body>, out: &T) -> ResponseResult {
let (mut resp, writer) = http_serve::streaming_body(&req).build();
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
resp.headers_mut().insert(
header::CONTENT_TYPE,
HeaderValue::from_static("application/json"),
);
read-only signals support (#28) This is mostly untested and useless by itself, but it's a starting point. In particular: * there's no way to set up signals or add/remove/update events yet except by manual changes to the database. * if you associate a signal with a camera then remove the camera, hitting /api/ will error out.
2019-06-06 16:18:13 -07:00
if let Some(mut w) = writer {
serde_json::to_writer(&mut w, out).map_err(internal_server_err)?;
}
Ok(resp)
}
more readable async web code This uses "async fn" throughout rather than a mix of async and the older futures style. And it takes advantage of the "self: Arc<Self>" syntax to avoid having a ServiceInner. It was confusing to have some methods on Service and some on ServiceInner; now that distinction is gone. One downside is there's a little more atomic reference-counting. Before, service_fn essentially took an &Arc<Self>, which means it could call Arc::clone where its use of self actually outlived the future (see stream_live_m4s) but didn't need to otherwise. After, it calls an async fn that takes Arc<Self>. Using &Arc<Self> is apparently possible (as of Rust 1.41) but using that with "async fn" means the returned future is tied to its lifetime. The workaround is to use async blocks as described here: <https://rust-lang.github.io/async-book/03_async_await/01_chapter.html> but that's really ugly: it brings back the explicit Future reference, requires futures::future::Either in some cases, and introduces another level of indenting. I think it's better to just pay the arc costs which are probably negligible, or at least cheaper than the boxing was before. Oh, and I make this compile on Rust 1.40 again as it claimed to. http-serve accidentally used the &Arc<Self> thing which broke this. Update to a freshly-pushed commit which doesn't do that.
2020-05-30 19:08:54 -07:00
fn csrf_matches(csrf: &str, session: auth::SessionHash) -> bool {
let mut b64 = [0u8; 32];
session.encode_base64(&mut b64);
::ring::constant_time::verify_slices_are_equal(&b64[..], csrf.as_bytes()).is_ok()
}
add a basic Javascript UI The Javascript is pretty amateurish I'm sure but at least it's something to iterate from. It's already much more pleasant for browsing through videos in several ways: * more responsive to load only a day at a time rather than 90+ days * much easier to see the same time segment on several cameras * more pleasant to have the videos load as a popup rather than a link that blows away your position in an enormous list * exposes the fancier .mp4 generation options: splitting at lengths other than the default, trimming to an arbitrary start and end time, including a subtitle track with timestamps. There's a slight regression in functionality: I didn't match the former top-level page which showed how much camera used of its disk allocation and the total duration of video. This is exposed in the JSON API, so it shouldn't be too hard to add back.
2017-10-21 21:54:27 -07:00
more readable async web code This uses "async fn" throughout rather than a mix of async and the older futures style. And it takes advantage of the "self: Arc<Self>" syntax to avoid having a ServiceInner. It was confusing to have some methods on Service and some on ServiceInner; now that distinction is gone. One downside is there's a little more atomic reference-counting. Before, service_fn essentially took an &Arc<Self>, which means it could call Arc::clone where its use of self actually outlived the future (see stream_live_m4s) but didn't need to otherwise. After, it calls an async fn that takes Arc<Self>. Using &Arc<Self> is apparently possible (as of Rust 1.41) but using that with "async fn" means the returned future is tied to its lifetime. The workaround is to use async blocks as described here: <https://rust-lang.github.io/async-book/03_async_await/01_chapter.html> but that's really ugly: it brings back the explicit Future reference, requires futures::future::Either in some cases, and introduces another level of indenting. I think it's better to just pay the arc costs which are probably negligible, or at least cheaper than the boxing was before. Oh, and I make this compile on Rust 1.40 again as it claimed to. http-serve accidentally used the &Arc<Self> thing which broke this. Update to a freshly-pushed commit which doesn't do that.
2020-05-30 19:08:54 -07:00
/// Extracts `s` cookie from the HTTP request. Does not authenticate.
fn extract_sid(req: &Request<hyper::Body>) -> Option<auth::RawSessionId> {
let hdr = match req.headers().get(header::COOKIE) {
None => return None,
Some(c) => c,
};
for mut cookie in hdr.as_bytes().split(|&b| b == b';') {
if cookie.starts_with(b" ") {
cookie = &cookie[1..];
}
if cookie.starts_with(b"s=") {
let s = &cookie[2..];
if let Ok(s) = auth::RawSessionId::decode_base64(s) {
return Some(s);
add concept of user/session permissions (I also considered the names "capabilities" and "scopes", but I think "permissions" is the most widely understood.) This is increasingly necessary as the web API becomes more capable. Among other things, it allows: * non-administrator users who can view but not access camera passwords or change any state * workers that update signal state based on cameras' built-in motion detection or a security system's events but don't need to view videos * control over what can be done without authenticating Currently session permissions are just copied from user permissions, but you can also imagine admin sessions vs not, as a checkbox when signing in. This would match the standard Unix workflow of using a non-administrative session most of the time. Relevant to my current signals work (#28) and to the addition of an administrative API (#35, including #66).
2019-06-19 15:17:50 -07:00
}
}
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 14:34:00 -08:00
}
more readable async web code This uses "async fn" throughout rather than a mix of async and the older futures style. And it takes advantage of the "self: Arc<Self>" syntax to avoid having a ServiceInner. It was confusing to have some methods on Service and some on ServiceInner; now that distinction is gone. One downside is there's a little more atomic reference-counting. Before, service_fn essentially took an &Arc<Self>, which means it could call Arc::clone where its use of self actually outlived the future (see stream_live_m4s) but didn't need to otherwise. After, it calls an async fn that takes Arc<Self>. Using &Arc<Self> is apparently possible (as of Rust 1.41) but using that with "async fn" means the returned future is tied to its lifetime. The workaround is to use async blocks as described here: <https://rust-lang.github.io/async-book/03_async_await/01_chapter.html> but that's really ugly: it brings back the explicit Future reference, requires futures::future::Either in some cases, and introduces another level of indenting. I think it's better to just pay the arc costs which are probably negligible, or at least cheaper than the boxing was before. Oh, and I make this compile on Rust 1.40 again as it claimed to. http-serve accidentally used the &Arc<Self> thing which broke this. Update to a freshly-pushed commit which doesn't do that.
2020-05-30 19:08:54 -07:00
None
}
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 14:34:00 -08:00
more readable async web code This uses "async fn" throughout rather than a mix of async and the older futures style. And it takes advantage of the "self: Arc<Self>" syntax to avoid having a ServiceInner. It was confusing to have some methods on Service and some on ServiceInner; now that distinction is gone. One downside is there's a little more atomic reference-counting. Before, service_fn essentially took an &Arc<Self>, which means it could call Arc::clone where its use of self actually outlived the future (see stream_live_m4s) but didn't need to otherwise. After, it calls an async fn that takes Arc<Self>. Using &Arc<Self> is apparently possible (as of Rust 1.41) but using that with "async fn" means the returned future is tied to its lifetime. The workaround is to use async blocks as described here: <https://rust-lang.github.io/async-book/03_async_await/01_chapter.html> but that's really ugly: it brings back the explicit Future reference, requires futures::future::Either in some cases, and introduces another level of indenting. I think it's better to just pay the arc costs which are probably negligible, or at least cheaper than the boxing was before. Oh, and I make this compile on Rust 1.40 again as it claimed to. http-serve accidentally used the &Arc<Self> thing which broke this. Update to a freshly-pushed commit which doesn't do that.
2020-05-30 19:08:54 -07:00
/// Extracts an `application/json` POST body from a request.
///
/// This returns the request body as bytes rather than performing
/// deserialization. Keeping the bytes allows the caller to use a `Deserialize`
/// that borrows from the bytes.
small readability improvements to web.rs I think this has a minor behavior change: permission denied replies change to HTTP 403 where they were HTTP 401. The new behavior seems more correct, as these errors can occur when authentication has succeeded but the session in question is not authorized for the given operation. The UI currently doesn't care about this distinction.
2021-03-10 16:18:47 -08:00
async fn extract_json_body(req: &mut Request<hyper::Body>) -> Result<Bytes, HttpError> {
UI preferences: #153 #155
2021-09-01 15:01:42 -07:00
if *req.method() != Method::POST {
small readability improvements to web.rs I think this has a minor behavior change: permission denied replies change to HTTP 403 where they were HTTP 401. The new behavior seems more correct, as these errors can occur when authentication has succeeded but the session in question is not authorized for the given operation. The UI currently doesn't care about this distinction.
2021-03-10 16:18:47 -08:00
return Err(plain_response(StatusCode::METHOD_NOT_ALLOWED, "POST expected").into());
more readable async web code This uses "async fn" throughout rather than a mix of async and the older futures style. And it takes advantage of the "self: Arc<Self>" syntax to avoid having a ServiceInner. It was confusing to have some methods on Service and some on ServiceInner; now that distinction is gone. One downside is there's a little more atomic reference-counting. Before, service_fn essentially took an &Arc<Self>, which means it could call Arc::clone where its use of self actually outlived the future (see stream_live_m4s) but didn't need to otherwise. After, it calls an async fn that takes Arc<Self>. Using &Arc<Self> is apparently possible (as of Rust 1.41) but using that with "async fn" means the returned future is tied to its lifetime. The workaround is to use async blocks as described here: <https://rust-lang.github.io/async-book/03_async_await/01_chapter.html> but that's really ugly: it brings back the explicit Future reference, requires futures::future::Either in some cases, and introduces another level of indenting. I think it's better to just pay the arc costs which are probably negligible, or at least cheaper than the boxing was before. Oh, and I make this compile on Rust 1.40 again as it claimed to. http-serve accidentally used the &Arc<Self> thing which broke this. Update to a freshly-pushed commit which doesn't do that.
2020-05-30 19:08:54 -07:00
}
let correct_mime_type = match req.headers().get(header::CONTENT_TYPE) {
Some(t) if t == "application/json" => true,
Some(t) if t == "application/json; charset=UTF-8" => true,
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
_ => false,
more readable async web code This uses "async fn" throughout rather than a mix of async and the older futures style. And it takes advantage of the "self: Arc<Self>" syntax to avoid having a ServiceInner. It was confusing to have some methods on Service and some on ServiceInner; now that distinction is gone. One downside is there's a little more atomic reference-counting. Before, service_fn essentially took an &Arc<Self>, which means it could call Arc::clone where its use of self actually outlived the future (see stream_live_m4s) but didn't need to otherwise. After, it calls an async fn that takes Arc<Self>. Using &Arc<Self> is apparently possible (as of Rust 1.41) but using that with "async fn" means the returned future is tied to its lifetime. The workaround is to use async blocks as described here: <https://rust-lang.github.io/async-book/03_async_await/01_chapter.html> but that's really ugly: it brings back the explicit Future reference, requires futures::future::Either in some cases, and introduces another level of indenting. I think it's better to just pay the arc costs which are probably negligible, or at least cheaper than the boxing was before. Oh, and I make this compile on Rust 1.40 again as it claimed to. http-serve accidentally used the &Arc<Self> thing which broke this. Update to a freshly-pushed commit which doesn't do that.
2020-05-30 19:08:54 -07:00
};
if !correct_mime_type {
return Err(bad_req("expected application/json request body"));
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 14:34:00 -08:00
}
more readable async web code This uses "async fn" throughout rather than a mix of async and the older futures style. And it takes advantage of the "self: Arc<Self>" syntax to avoid having a ServiceInner. It was confusing to have some methods on Service and some on ServiceInner; now that distinction is gone. One downside is there's a little more atomic reference-counting. Before, service_fn essentially took an &Arc<Self>, which means it could call Arc::clone where its use of self actually outlived the future (see stream_live_m4s) but didn't need to otherwise. After, it calls an async fn that takes Arc<Self>. Using &Arc<Self> is apparently possible (as of Rust 1.41) but using that with "async fn" means the returned future is tied to its lifetime. The workaround is to use async blocks as described here: <https://rust-lang.github.io/async-book/03_async_await/01_chapter.html> but that's really ugly: it brings back the explicit Future reference, requires futures::future::Either in some cases, and introduces another level of indenting. I think it's better to just pay the arc costs which are probably negligible, or at least cheaper than the boxing was before. Oh, and I make this compile on Rust 1.40 again as it claimed to. http-serve accidentally used the &Arc<Self> thing which broke this. Update to a freshly-pushed commit which doesn't do that.
2020-05-30 19:08:54 -07:00
let b = ::std::mem::replace(req.body_mut(), hyper::Body::empty());
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
hyper::body::to_bytes(b)
.await
more readable async web code This uses "async fn" throughout rather than a mix of async and the older futures style. And it takes advantage of the "self: Arc<Self>" syntax to avoid having a ServiceInner. It was confusing to have some methods on Service and some on ServiceInner; now that distinction is gone. One downside is there's a little more atomic reference-counting. Before, service_fn essentially took an &Arc<Self>, which means it could call Arc::clone where its use of self actually outlived the future (see stream_live_m4s) but didn't need to otherwise. After, it calls an async fn that takes Arc<Self>. Using &Arc<Self> is apparently possible (as of Rust 1.41) but using that with "async fn" means the returned future is tied to its lifetime. The workaround is to use async blocks as described here: <https://rust-lang.github.io/async-book/03_async_await/01_chapter.html> but that's really ugly: it brings back the explicit Future reference, requires futures::future::Either in some cases, and introduces another level of indenting. I think it's better to just pay the arc costs which are probably negligible, or at least cheaper than the boxing was before. Oh, and I make this compile on Rust 1.40 again as it claimed to. http-serve accidentally used the &Arc<Self> thing which broke this. Update to a freshly-pushed commit which doesn't do that.
2020-05-30 19:08:54 -07:00
.map_err(|e| internal_server_err(format_err!("unable to read request body: {}", e)))
}
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 14:34:00 -08:00
more readable async web code This uses "async fn" throughout rather than a mix of async and the older futures style. And it takes advantage of the "self: Arc<Self>" syntax to avoid having a ServiceInner. It was confusing to have some methods on Service and some on ServiceInner; now that distinction is gone. One downside is there's a little more atomic reference-counting. Before, service_fn essentially took an &Arc<Self>, which means it could call Arc::clone where its use of self actually outlived the future (see stream_live_m4s) but didn't need to otherwise. After, it calls an async fn that takes Arc<Self>. Using &Arc<Self> is apparently possible (as of Rust 1.41) but using that with "async fn" means the returned future is tied to its lifetime. The workaround is to use async blocks as described here: <https://rust-lang.github.io/async-book/03_async_await/01_chapter.html> but that's really ugly: it brings back the explicit Future reference, requires futures::future::Either in some cases, and introduces another level of indenting. I think it's better to just pay the arc costs which are probably negligible, or at least cheaper than the boxing was before. Oh, and I make this compile on Rust 1.40 again as it claimed to. http-serve accidentally used the &Arc<Self> thing which broke this. Update to a freshly-pushed commit which doesn't do that.
2020-05-30 19:08:54 -07:00
pub struct Config<'a> {
pub db: Arc<db::Database>,
pub ui_dir: Option<&'a std::path::Path>,
pub trust_forward_hdrs: bool,
pub time_zone_name: String,
pub allow_unauthenticated_permissions: Option<db::Permissions>,
}
pub struct Service {
db: Arc<db::Database>,
ui_dir: Option<Arc<FsDir>>,
dirs_by_stream_id: Arc<FnvHashMap<i32, Arc<SampleFileDir>>>,
time_zone_name: String,
allow_unauthenticated_permissions: Option<db::Permissions>,
trust_forward_hdrs: bool,
}
/// Useful HTTP `Cache-Control` values to set on successful (HTTP 200) API responses.
enum CacheControl {
/// For endpoints which have private data that may change from request to request.
PrivateDynamic,
/// For endpoints which rarely change for a given URL.
/// E.g., a fixed segment of video. The underlying video logically never changes; there may
/// rarely be some software change to the actual bytes (which would result in a new etag) so
/// (unlike the content-hashed static content) it's not entirely immutable.
PrivateStatic,
None,
}
impl Service {
pub fn new(config: Config) -> Result<Self, Error> {
let mut ui_dir = None;
if let Some(d) = config.ui_dir {
match FsDir::builder().for_path(&d) {
Err(e) => {
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
warn!(
"Unable to load --ui-dir={}; will serve no static files: {}",
d.display(),
e
);
}
more readable async web code This uses "async fn" throughout rather than a mix of async and the older futures style. And it takes advantage of the "self: Arc<Self>" syntax to avoid having a ServiceInner. It was confusing to have some methods on Service and some on ServiceInner; now that distinction is gone. One downside is there's a little more atomic reference-counting. Before, service_fn essentially took an &Arc<Self>, which means it could call Arc::clone where its use of self actually outlived the future (see stream_live_m4s) but didn't need to otherwise. After, it calls an async fn that takes Arc<Self>. Using &Arc<Self> is apparently possible (as of Rust 1.41) but using that with "async fn" means the returned future is tied to its lifetime. The workaround is to use async blocks as described here: <https://rust-lang.github.io/async-book/03_async_await/01_chapter.html> but that's really ugly: it brings back the explicit Future reference, requires futures::future::Either in some cases, and introduces another level of indenting. I think it's better to just pay the arc costs which are probably negligible, or at least cheaper than the boxing was before. Oh, and I make this compile on Rust 1.40 again as it claimed to. http-serve accidentally used the &Arc<Self> thing which broke this. Update to a freshly-pushed commit which doesn't do that.
2020-05-30 19:08:54 -07:00
Ok(d) => ui_dir = Some(d),
};
}
let dirs_by_stream_id = {
let l = config.db.lock();
let mut d =
FnvHashMap::with_capacity_and_hasher(l.streams_by_id().len(), Default::default());
for (&id, s) in l.streams_by_id().iter() {
let dir_id = match s.sample_file_dir_id {
Some(d) => d,
None => continue,
support run splitting in json api
2017-10-17 09:00:05 -07:00
};
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
d.insert(id, l.sample_file_dirs_by_id().get(&dir_id).unwrap().get()?);
support run splitting in json api
2017-10-17 09:00:05 -07:00
}
more readable async web code This uses "async fn" throughout rather than a mix of async and the older futures style. And it takes advantage of the "self: Arc<Self>" syntax to avoid having a ServiceInner. It was confusing to have some methods on Service and some on ServiceInner; now that distinction is gone. One downside is there's a little more atomic reference-counting. Before, service_fn essentially took an &Arc<Self>, which means it could call Arc::clone where its use of self actually outlived the future (see stream_live_m4s) but didn't need to otherwise. After, it calls an async fn that takes Arc<Self>. Using &Arc<Self> is apparently possible (as of Rust 1.41) but using that with "async fn" means the returned future is tied to its lifetime. The workaround is to use async blocks as described here: <https://rust-lang.github.io/async-book/03_async_await/01_chapter.html> but that's really ugly: it brings back the explicit Future reference, requires futures::future::Either in some cases, and introduces another level of indenting. I think it's better to just pay the arc costs which are probably negligible, or at least cheaper than the boxing was before. Oh, and I make this compile on Rust 1.40 again as it claimed to. http-serve accidentally used the &Arc<Self> thing which broke this. Update to a freshly-pushed commit which doesn't do that.
2020-05-30 19:08:54 -07:00
Arc::new(d)
reorganize /recordings JSON response I want to start returning the pixel aspect ratio of each video sample entry. It's silly to duplicate it for each returned recording, so let's instead return a videoSampleEntryId and then put all the information about each VSE once. This change doesn't actually handle pixel aspect ratio server-side yet. Most likely I'll require a new schema version for that, to store it as a new column in the database. Codec-specific logic in the database layer is awkward and I'd like to avoid it. I did a similar schema change to add the rfc6381_codec. I also adjusted ui-src/lib/models/Recording.js in a few ways: * fixed a couple mismatches between its field name and the key defined in the API. Consistency aids understanding. * dropped all the getters in favor of just setting the fields (with type annotations) as described here: https://google.github.io/styleguide/jsguide.html#features-classes-fields * where the wire format used undefined (to save space), translate it to a more natural null or false.
2020-03-13 21:20:51 -07:00
};
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 14:34:00 -08:00
more readable async web code This uses "async fn" throughout rather than a mix of async and the older futures style. And it takes advantage of the "self: Arc<Self>" syntax to avoid having a ServiceInner. It was confusing to have some methods on Service and some on ServiceInner; now that distinction is gone. One downside is there's a little more atomic reference-counting. Before, service_fn essentially took an &Arc<Self>, which means it could call Arc::clone where its use of self actually outlived the future (see stream_live_m4s) but didn't need to otherwise. After, it calls an async fn that takes Arc<Self>. Using &Arc<Self> is apparently possible (as of Rust 1.41) but using that with "async fn" means the returned future is tied to its lifetime. The workaround is to use async blocks as described here: <https://rust-lang.github.io/async-book/03_async_await/01_chapter.html> but that's really ugly: it brings back the explicit Future reference, requires futures::future::Either in some cases, and introduces another level of indenting. I think it's better to just pay the arc costs which are probably negligible, or at least cheaper than the boxing was before. Oh, and I make this compile on Rust 1.40 again as it claimed to. http-serve accidentally used the &Arc<Self> thing which broke this. Update to a freshly-pushed commit which doesn't do that.
2020-05-30 19:08:54 -07:00
Ok(Service {
db: config.db,
dirs_by_stream_id,
ui_dir,
allow_unauthenticated_permissions: config.allow_unauthenticated_permissions,
trust_forward_hdrs: config.trust_forward_hdrs,
time_zone_name: config.time_zone_name,
})
support segmented mp4s This is intended to support HTML5 Media Source Extensions, which I expect to be the most practical way to make a good web UI with a proper scrub bar and such. This feature has had very limited testing on Chrome and Firefox, and that was not entirely successful. More work is needed before it's usable, but this seems like a helpful progress checkpoint.
2017-10-01 15:29:22 -07:00
}
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
fn stream_live_m4s(
self: Arc<Self>,
req: Request<::hyper::Body>,
caller: Caller,
uuid: Uuid,
stream_type: db::StreamType,
) -> ResponseResult {
add concept of user/session permissions (I also considered the names "capabilities" and "scopes", but I think "permissions" is the most widely understood.) This is increasingly necessary as the web API becomes more capable. Among other things, it allows: * non-administrator users who can view but not access camera passwords or change any state * workers that update signal state based on cameras' built-in motion detection or a security system's events but don't need to view videos * control over what can be done without authenticating Currently session permissions are just copied from user permissions, but you can also imagine admin sessions vs not, as a checkbox when signing in. This would match the standard Unix workflow of using a non-administrative session most of the time. Relevant to my current signals work (#28) and to the addition of an administrative API (#35, including #66).
2019-06-19 15:17:50 -07:00
if !caller.permissions.view_video {
small readability improvements to web.rs I think this has a minor behavior change: permission denied replies change to HTTP 403 where they were HTTP 401. The new behavior seems more correct, as these errors can occur when authentication has succeeded but the session in question is not authorized for the given operation. The UI currently doesn't care about this distinction.
2021-03-10 16:18:47 -08:00
bail_t!(PermissionDenied, "view_video required");
add concept of user/session permissions (I also considered the names "capabilities" and "scopes", but I think "permissions" is the most widely understood.) This is increasingly necessary as the web API becomes more capable. Among other things, it allows: * non-administrator users who can view but not access camera passwords or change any state * workers that update signal state based on cameras' built-in motion detection or a security system's events but don't need to view videos * control over what can be done without authenticating Currently session permissions are just copied from user permissions, but you can also imagine admin sessions vs not, as a checkbox when signing in. This would match the standard Unix workflow of using a non-administrative session most of the time. Relevant to my current signals work (#28) and to the addition of an administrative API (#35, including #66).
2019-06-19 15:17:50 -07:00
}
more readable async web code This uses "async fn" throughout rather than a mix of async and the older futures style. And it takes advantage of the "self: Arc<Self>" syntax to avoid having a ServiceInner. It was confusing to have some methods on Service and some on ServiceInner; now that distinction is gone. One downside is there's a little more atomic reference-counting. Before, service_fn essentially took an &Arc<Self>, which means it could call Arc::clone where its use of self actually outlived the future (see stream_live_m4s) but didn't need to otherwise. After, it calls an async fn that takes Arc<Self>. Using &Arc<Self> is apparently possible (as of Rust 1.41) but using that with "async fn" means the returned future is tied to its lifetime. The workaround is to use async blocks as described here: <https://rust-lang.github.io/async-book/03_async_await/01_chapter.html> but that's really ugly: it brings back the explicit Future reference, requires futures::future::Either in some cases, and introduces another level of indenting. I think it's better to just pay the arc costs which are probably negligible, or at least cheaper than the boxing was before. Oh, and I make this compile on Rust 1.40 again as it claimed to. http-serve accidentally used the &Arc<Self> thing which broke this. Update to a freshly-pushed commit which doesn't do that.
2020-05-30 19:08:54 -07:00
let stream_id;
let open_id;
let (sub_tx, sub_rx) = futures::channel::mpsc::unbounded();
give downloaded .mp4s a useful filename This is effective both for Chrome's "Save As" dialog and for curl -OJ. It makes the filename like 20190717135519-driveway-main.mp4 rather than view.mp4 (Chrome) or view.mp4?s=33-36&ts=true (Curl).
2020-02-16 23:16:19 -08:00
{
more readable async web code This uses "async fn" throughout rather than a mix of async and the older futures style. And it takes advantage of the "self: Arc<Self>" syntax to avoid having a ServiceInner. It was confusing to have some methods on Service and some on ServiceInner; now that distinction is gone. One downside is there's a little more atomic reference-counting. Before, service_fn essentially took an &Arc<Self>, which means it could call Arc::clone where its use of self actually outlived the future (see stream_live_m4s) but didn't need to otherwise. After, it calls an async fn that takes Arc<Self>. Using &Arc<Self> is apparently possible (as of Rust 1.41) but using that with "async fn" means the returned future is tied to its lifetime. The workaround is to use async blocks as described here: <https://rust-lang.github.io/async-book/03_async_await/01_chapter.html> but that's really ugly: it brings back the explicit Future reference, requires futures::future::Either in some cases, and introduces another level of indenting. I think it's better to just pay the arc costs which are probably negligible, or at least cheaper than the boxing was before. Oh, and I make this compile on Rust 1.40 again as it claimed to. http-serve accidentally used the &Arc<Self> thing which broke this. Update to a freshly-pushed commit which doesn't do that.
2020-05-30 19:08:54 -07:00
let mut db = self.db.lock();
open_id = match db.open {
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
None => {
small readability improvements to web.rs I think this has a minor behavior change: permission denied replies change to HTTP 403 where they were HTTP 401. The new behavior seems more correct, as these errors can occur when authentication has succeeded but the session in question is not authorized for the given operation. The UI currently doesn't care about this distinction.
2021-03-10 16:18:47 -08:00
bail_t!(
FailedPrecondition,
"database is read-only; there are no live streams"
);
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
}
more readable async web code This uses "async fn" throughout rather than a mix of async and the older futures style. And it takes advantage of the "self: Arc<Self>" syntax to avoid having a ServiceInner. It was confusing to have some methods on Service and some on ServiceInner; now that distinction is gone. One downside is there's a little more atomic reference-counting. Before, service_fn essentially took an &Arc<Self>, which means it could call Arc::clone where its use of self actually outlived the future (see stream_live_m4s) but didn't need to otherwise. After, it calls an async fn that takes Arc<Self>. Using &Arc<Self> is apparently possible (as of Rust 1.41) but using that with "async fn" means the returned future is tied to its lifetime. The workaround is to use async blocks as described here: <https://rust-lang.github.io/async-book/03_async_await/01_chapter.html> but that's really ugly: it brings back the explicit Future reference, requires futures::future::Either in some cases, and introduces another level of indenting. I think it's better to just pay the arc costs which are probably negligible, or at least cheaper than the boxing was before. Oh, and I make this compile on Rust 1.40 again as it claimed to. http-serve accidentally used the &Arc<Self> thing which broke this. Update to a freshly-pushed commit which doesn't do that.
2020-05-30 19:08:54 -07:00
Some(o) => o.id,
};
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
let camera = db.get_camera(uuid).ok_or_else(|| {
plain_response(StatusCode::NOT_FOUND, format!("no such camera {}", uuid))
})?;
stream_id = camera.streams[stream_type.index()].ok_or_else(|| {
small readability improvements to web.rs I think this has a minor behavior change: permission denied replies change to HTTP 403 where they were HTTP 401. The new behavior seems more correct, as these errors can occur when authentication has succeeded but the session in question is not authorized for the given operation. The UI currently doesn't care about this distinction.
2021-03-10 16:18:47 -08:00
format_err_t!(NotFound, "no such stream {}/{}", uuid, stream_type)
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
})?;
db.watch_live(
stream_id,
Box::new(move |l| sub_tx.unbounded_send(l).is_ok()),
)
.expect("stream_id refed by camera");
more readable async web code This uses "async fn" throughout rather than a mix of async and the older futures style. And it takes advantage of the "self: Arc<Self>" syntax to avoid having a ServiceInner. It was confusing to have some methods on Service and some on ServiceInner; now that distinction is gone. One downside is there's a little more atomic reference-counting. Before, service_fn essentially took an &Arc<Self>, which means it could call Arc::clone where its use of self actually outlived the future (see stream_live_m4s) but didn't need to otherwise. After, it calls an async fn that takes Arc<Self>. Using &Arc<Self> is apparently possible (as of Rust 1.41) but using that with "async fn" means the returned future is tied to its lifetime. The workaround is to use async blocks as described here: <https://rust-lang.github.io/async-book/03_async_await/01_chapter.html> but that's really ugly: it brings back the explicit Future reference, requires futures::future::Either in some cases, and introduces another level of indenting. I think it's better to just pay the arc costs which are probably negligible, or at least cheaper than the boxing was before. Oh, and I make this compile on Rust 1.40 again as it claimed to. http-serve accidentally used the &Arc<Self> thing which broke this. Update to a freshly-pushed commit which doesn't do that.
2020-05-30 19:08:54 -07:00
}
upgrade tokio, bytes, hyper, reqwest, http-serve This was mostly straightforward. The most confusing part waas the Sync bound change on body streams. I copied what hyper did and it seemed to work. /shruggie
2021-01-27 11:47:52 -08:00
let response =
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
tungstenite::handshake::server::create_response_with_body(&req, hyper::Body::empty)
.map_err(|e| bad_req(e.to_string()))?;
upgrade tokio, bytes, hyper, reqwest, http-serve This was mostly straightforward. The most confusing part waas the Sync bound change on body streams. I copied what hyper did and it seemed to work. /shruggie
2021-01-27 11:47:52 -08:00
let (parts, _) = response.into_parts();
more readable async web code This uses "async fn" throughout rather than a mix of async and the older futures style. And it takes advantage of the "self: Arc<Self>" syntax to avoid having a ServiceInner. It was confusing to have some methods on Service and some on ServiceInner; now that distinction is gone. One downside is there's a little more atomic reference-counting. Before, service_fn essentially took an &Arc<Self>, which means it could call Arc::clone where its use of self actually outlived the future (see stream_live_m4s) but didn't need to otherwise. After, it calls an async fn that takes Arc<Self>. Using &Arc<Self> is apparently possible (as of Rust 1.41) but using that with "async fn" means the returned future is tied to its lifetime. The workaround is to use async blocks as described here: <https://rust-lang.github.io/async-book/03_async_await/01_chapter.html> but that's really ugly: it brings back the explicit Future reference, requires futures::future::Either in some cases, and introduces another level of indenting. I think it's better to just pay the arc costs which are probably negligible, or at least cheaper than the boxing was before. Oh, and I make this compile on Rust 1.40 again as it claimed to. http-serve accidentally used the &Arc<Self> thing which broke this. Update to a freshly-pushed commit which doesn't do that.
2020-05-30 19:08:54 -07:00
upgrade tokio, bytes, hyper, reqwest, http-serve This was mostly straightforward. The most confusing part waas the Sync bound change on body streams. I copied what hyper did and it seemed to work. /shruggie
2021-01-27 11:47:52 -08:00
tokio::spawn(self.stream_live_m4s_ws(stream_id, open_id, req, sub_rx));
more readable async web code This uses "async fn" throughout rather than a mix of async and the older futures style. And it takes advantage of the "self: Arc<Self>" syntax to avoid having a ServiceInner. It was confusing to have some methods on Service and some on ServiceInner; now that distinction is gone. One downside is there's a little more atomic reference-counting. Before, service_fn essentially took an &Arc<Self>, which means it could call Arc::clone where its use of self actually outlived the future (see stream_live_m4s) but didn't need to otherwise. After, it calls an async fn that takes Arc<Self>. Using &Arc<Self> is apparently possible (as of Rust 1.41) but using that with "async fn" means the returned future is tied to its lifetime. The workaround is to use async blocks as described here: <https://rust-lang.github.io/async-book/03_async_await/01_chapter.html> but that's really ugly: it brings back the explicit Future reference, requires futures::future::Either in some cases, and introduces another level of indenting. I think it's better to just pay the arc costs which are probably negligible, or at least cheaper than the boxing was before. Oh, and I make this compile on Rust 1.40 again as it claimed to. http-serve accidentally used the &Arc<Self> thing which broke this. Update to a freshly-pushed commit which doesn't do that.
2020-05-30 19:08:54 -07:00
Ok(Response::from_parts(parts, Body::from("")))
}
give downloaded .mp4s a useful filename This is effective both for Chrome's "Save As" dialog and for curl -OJ. It makes the filename like 20190717135519-driveway-main.mp4 rather than view.mp4 (Chrome) or view.mp4?s=33-36&ts=true (Curl).
2020-02-16 23:16:19 -08:00
more readable async web code This uses "async fn" throughout rather than a mix of async and the older futures style. And it takes advantage of the "self: Arc<Self>" syntax to avoid having a ServiceInner. It was confusing to have some methods on Service and some on ServiceInner; now that distinction is gone. One downside is there's a little more atomic reference-counting. Before, service_fn essentially took an &Arc<Self>, which means it could call Arc::clone where its use of self actually outlived the future (see stream_live_m4s) but didn't need to otherwise. After, it calls an async fn that takes Arc<Self>. Using &Arc<Self> is apparently possible (as of Rust 1.41) but using that with "async fn" means the returned future is tied to its lifetime. The workaround is to use async blocks as described here: <https://rust-lang.github.io/async-book/03_async_await/01_chapter.html> but that's really ugly: it brings back the explicit Future reference, requires futures::future::Either in some cases, and introduces another level of indenting. I think it's better to just pay the arc costs which are probably negligible, or at least cheaper than the boxing was before. Oh, and I make this compile on Rust 1.40 again as it claimed to. http-serve accidentally used the &Arc<Self> thing which broke this. Update to a freshly-pushed commit which doesn't do that.
2020-05-30 19:08:54 -07:00
async fn stream_live_m4s_ws(
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
self: Arc<Self>,
stream_id: i32,
open_id: u32,
req: hyper::Request<hyper::Body>,
send keepalives on live.m4s Chrome appears to time out at 60 seconds of inactivity otherwise. I think it's better to keep the stream open, even if the camera is broken. The implementation looks awkward, but that might be the state of Rust async right now.
2021-03-25 23:01:38 -07:00
sub_rx: futures::channel::mpsc::UnboundedReceiver<db::LiveSegment>,
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
) {
upgrade tokio, bytes, hyper, reqwest, http-serve This was mostly straightforward. The most confusing part waas the Sync bound change on body streams. I copied what hyper did and it seemed to work. /shruggie
2021-01-27 11:47:52 -08:00
let upgraded = match hyper::upgrade::on(req).await {
more readable async web code This uses "async fn" throughout rather than a mix of async and the older futures style. And it takes advantage of the "self: Arc<Self>" syntax to avoid having a ServiceInner. It was confusing to have some methods on Service and some on ServiceInner; now that distinction is gone. One downside is there's a little more atomic reference-counting. Before, service_fn essentially took an &Arc<Self>, which means it could call Arc::clone where its use of self actually outlived the future (see stream_live_m4s) but didn't need to otherwise. After, it calls an async fn that takes Arc<Self>. Using &Arc<Self> is apparently possible (as of Rust 1.41) but using that with "async fn" means the returned future is tied to its lifetime. The workaround is to use async blocks as described here: <https://rust-lang.github.io/async-book/03_async_await/01_chapter.html> but that's really ugly: it brings back the explicit Future reference, requires futures::future::Either in some cases, and introduces another level of indenting. I think it's better to just pay the arc costs which are probably negligible, or at least cheaper than the boxing was before. Oh, and I make this compile on Rust 1.40 again as it claimed to. http-serve accidentally used the &Arc<Self> thing which broke this. Update to a freshly-pushed commit which doesn't do that.
2020-05-30 19:08:54 -07:00
Ok(u) => u,
Err(e) => {
warn!("Unable to upgrade stream to websocket: {}", e);
return;
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
}
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 14:34:00 -08:00
};
send keepalives on live.m4s Chrome appears to time out at 60 seconds of inactivity otherwise. I think it's better to keep the stream open, even if the camera is broken. The implementation looks awkward, but that might be the state of Rust async right now.
2021-03-25 23:01:38 -07:00
let ws = tokio_tungstenite::WebSocketStream::from_raw_socket(
more readable async web code This uses "async fn" throughout rather than a mix of async and the older futures style. And it takes advantage of the "self: Arc<Self>" syntax to avoid having a ServiceInner. It was confusing to have some methods on Service and some on ServiceInner; now that distinction is gone. One downside is there's a little more atomic reference-counting. Before, service_fn essentially took an &Arc<Self>, which means it could call Arc::clone where its use of self actually outlived the future (see stream_live_m4s) but didn't need to otherwise. After, it calls an async fn that takes Arc<Self>. Using &Arc<Self> is apparently possible (as of Rust 1.41) but using that with "async fn" means the returned future is tied to its lifetime. The workaround is to use async blocks as described here: <https://rust-lang.github.io/async-book/03_async_await/01_chapter.html> but that's really ugly: it brings back the explicit Future reference, requires futures::future::Either in some cases, and introduces another level of indenting. I think it's better to just pay the arc costs which are probably negligible, or at least cheaper than the boxing was before. Oh, and I make this compile on Rust 1.40 again as it claimed to. http-serve accidentally used the &Arc<Self> thing which broke this. Update to a freshly-pushed commit which doesn't do that.
2020-05-30 19:08:54 -07:00
upgraded,
tungstenite::protocol::Role::Server,
None,
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
)
.await;
live stream frame-by-frame rather than GOP-by-GOP (#59) This should reduce live stream latency by two seconds when my cameras are at their default setting (I frame interval = 2 * frame rate)! I was under the impression that every HTML5 Media Source Extensions media segment had to start with a Random Access Point. This used to be true, but apparently changed quite a while ago: https://bugs.chromium.org/p/chromium/issues/detail?id=229412 Support generating segments that don't start with a key frame, and plumb this through the mp4 media segment generation logic. Add some extra error checking in mp4 slice handling, as my first attempts had a mismatch between expected and actual lengths that silently returned corrupted .m4s files. Also pull everything from the most recent key frame on along with the first live segment to reduce startup latency. Live view is quite a bit more pleasant now.
2020-08-07 15:30:22 -07:00
send keepalives on live.m4s Chrome appears to time out at 60 seconds of inactivity otherwise. I think it's better to keep the stream open, even if the camera is broken. The implementation looks awkward, but that might be the state of Rust async right now.
2021-03-25 23:01:38 -07:00
if let Err(e) = self
.stream_live_m4s_ws_loop(stream_id, open_id, sub_rx, ws)
.await
{
info!("Dropping WebSocket after error: {}", e);
}
}
/// Helper for `stream_live_m4s_ws` that returns error when the stream is dropped.
/// The outer function logs the error.
async fn stream_live_m4s_ws_loop(
self: Arc<Self>,
stream_id: i32,
open_id: u32,
sub_rx: futures::channel::mpsc::UnboundedReceiver<db::LiveSegment>,
mut ws: tokio_tungstenite::WebSocketStream<hyper::upgrade::Upgraded>,
) -> Result<(), Error> {
let keepalive = tokio_stream::wrappers::IntervalStream::new(tokio::time::interval(
std::time::Duration::new(30, 0),
));
let mut combo = futures::stream::select(
address some no-op clippy warnings
2021-05-17 14:31:50 -07:00
sub_rx.map(Either::Left),
send keepalives on live.m4s Chrome appears to time out at 60 seconds of inactivity otherwise. I think it's better to keep the stream open, even if the camera is broken. The implementation looks awkward, but that might be the state of Rust async right now.
2021-03-25 23:01:38 -07:00
keepalive.map(|_| Either::Right(())),
);
// On the first LiveSegment, send all the data from the previous key frame onward.
// For LiveSegments, it's okay to send a single non-key frame at a time.
live stream frame-by-frame rather than GOP-by-GOP (#59) This should reduce live stream latency by two seconds when my cameras are at their default setting (I frame interval = 2 * frame rate)! I was under the impression that every HTML5 Media Source Extensions media segment had to start with a Random Access Point. This used to be true, but apparently changed quite a while ago: https://bugs.chromium.org/p/chromium/issues/detail?id=229412 Support generating segments that don't start with a key frame, and plumb this through the mp4 media segment generation logic. Add some extra error checking in mp4 slice handling, as my first attempts had a mismatch between expected and actual lengths that silently returned corrupted .m4s files. Also pull everything from the most recent key frame on along with the first live segment to reduce startup latency. Live view is quite a bit more pleasant now.
2020-08-07 15:30:22 -07:00
let mut start_at_key = true;
more readable async web code This uses "async fn" throughout rather than a mix of async and the older futures style. And it takes advantage of the "self: Arc<Self>" syntax to avoid having a ServiceInner. It was confusing to have some methods on Service and some on ServiceInner; now that distinction is gone. One downside is there's a little more atomic reference-counting. Before, service_fn essentially took an &Arc<Self>, which means it could call Arc::clone where its use of self actually outlived the future (see stream_live_m4s) but didn't need to otherwise. After, it calls an async fn that takes Arc<Self>. Using &Arc<Self> is apparently possible (as of Rust 1.41) but using that with "async fn" means the returned future is tied to its lifetime. The workaround is to use async blocks as described here: <https://rust-lang.github.io/async-book/03_async_await/01_chapter.html> but that's really ugly: it brings back the explicit Future reference, requires futures::future::Either in some cases, and introduces another level of indenting. I think it's better to just pay the arc costs which are probably negligible, or at least cheaper than the boxing was before. Oh, and I make this compile on Rust 1.40 again as it claimed to. http-serve accidentally used the &Arc<Self> thing which broke this. Update to a freshly-pushed commit which doesn't do that.
2020-05-30 19:08:54 -07:00
loop {
send keepalives on live.m4s Chrome appears to time out at 60 seconds of inactivity otherwise. I think it's better to keep the stream open, even if the camera is broken. The implementation looks awkward, but that might be the state of Rust async right now.
2021-03-25 23:01:38 -07:00
let next = combo
.next()
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
.await
send keepalives on live.m4s Chrome appears to time out at 60 seconds of inactivity otherwise. I think it's better to keep the stream open, even if the camera is broken. The implementation looks awkward, but that might be the state of Rust async right now.
2021-03-25 23:01:38 -07:00
.unwrap_or_else(|| unreachable!("timer stream never ends"));
match next {
Either::Left(live) => {
self.stream_live_m4s_chunk(open_id, stream_id, &mut ws, live, start_at_key)
.await?;
start_at_key = false;
}
Either::Right(_) => {
ws.send(tungstenite::Message::Ping(Vec::new())).await?;
}
more readable async web code This uses "async fn" throughout rather than a mix of async and the older futures style. And it takes advantage of the "self: Arc<Self>" syntax to avoid having a ServiceInner. It was confusing to have some methods on Service and some on ServiceInner; now that distinction is gone. One downside is there's a little more atomic reference-counting. Before, service_fn essentially took an &Arc<Self>, which means it could call Arc::clone where its use of self actually outlived the future (see stream_live_m4s) but didn't need to otherwise. After, it calls an async fn that takes Arc<Self>. Using &Arc<Self> is apparently possible (as of Rust 1.41) but using that with "async fn" means the returned future is tied to its lifetime. The workaround is to use async blocks as described here: <https://rust-lang.github.io/async-book/03_async_await/01_chapter.html> but that's really ugly: it brings back the explicit Future reference, requires futures::future::Either in some cases, and introduces another level of indenting. I think it's better to just pay the arc costs which are probably negligible, or at least cheaper than the boxing was before. Oh, and I make this compile on Rust 1.40 again as it claimed to. http-serve accidentally used the &Arc<Self> thing which broke this. Update to a freshly-pushed commit which doesn't do that.
2020-05-30 19:08:54 -07:00
}
}
}
send keepalives on live.m4s Chrome appears to time out at 60 seconds of inactivity otherwise. I think it's better to keep the stream open, even if the camera is broken. The implementation looks awkward, but that might be the state of Rust async right now.
2021-03-25 23:01:38 -07:00
/// Sends a single live segment chunk of a `live.m4s` stream.
more readable async web code This uses "async fn" throughout rather than a mix of async and the older futures style. And it takes advantage of the "self: Arc<Self>" syntax to avoid having a ServiceInner. It was confusing to have some methods on Service and some on ServiceInner; now that distinction is gone. One downside is there's a little more atomic reference-counting. Before, service_fn essentially took an &Arc<Self>, which means it could call Arc::clone where its use of self actually outlived the future (see stream_live_m4s) but didn't need to otherwise. After, it calls an async fn that takes Arc<Self>. Using &Arc<Self> is apparently possible (as of Rust 1.41) but using that with "async fn" means the returned future is tied to its lifetime. The workaround is to use async blocks as described here: <https://rust-lang.github.io/async-book/03_async_await/01_chapter.html> but that's really ugly: it brings back the explicit Future reference, requires futures::future::Either in some cases, and introduces another level of indenting. I think it's better to just pay the arc costs which are probably negligible, or at least cheaper than the boxing was before. Oh, and I make this compile on Rust 1.40 again as it claimed to. http-serve accidentally used the &Arc<Self> thing which broke this. Update to a freshly-pushed commit which doesn't do that.
2020-05-30 19:08:54 -07:00
async fn stream_live_m4s_chunk(
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
&self,
open_id: u32,
stream_id: i32,
more readable async web code This uses "async fn" throughout rather than a mix of async and the older futures style. And it takes advantage of the "self: Arc<Self>" syntax to avoid having a ServiceInner. It was confusing to have some methods on Service and some on ServiceInner; now that distinction is gone. One downside is there's a little more atomic reference-counting. Before, service_fn essentially took an &Arc<Self>, which means it could call Arc::clone where its use of self actually outlived the future (see stream_live_m4s) but didn't need to otherwise. After, it calls an async fn that takes Arc<Self>. Using &Arc<Self> is apparently possible (as of Rust 1.41) but using that with "async fn" means the returned future is tied to its lifetime. The workaround is to use async blocks as described here: <https://rust-lang.github.io/async-book/03_async_await/01_chapter.html> but that's really ugly: it brings back the explicit Future reference, requires futures::future::Either in some cases, and introduces another level of indenting. I think it's better to just pay the arc costs which are probably negligible, or at least cheaper than the boxing was before. Oh, and I make this compile on Rust 1.40 again as it claimed to. http-serve accidentally used the &Arc<Self> thing which broke this. Update to a freshly-pushed commit which doesn't do that.
2020-05-30 19:08:54 -07:00
ws: &mut tokio_tungstenite::WebSocketStream<hyper::upgrade::Upgraded>,
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
live: db::LiveSegment,
start_at_key: bool,
) -> Result<(), Error> {
more readable async web code This uses "async fn" throughout rather than a mix of async and the older futures style. And it takes advantage of the "self: Arc<Self>" syntax to avoid having a ServiceInner. It was confusing to have some methods on Service and some on ServiceInner; now that distinction is gone. One downside is there's a little more atomic reference-counting. Before, service_fn essentially took an &Arc<Self>, which means it could call Arc::clone where its use of self actually outlived the future (see stream_live_m4s) but didn't need to otherwise. After, it calls an async fn that takes Arc<Self>. Using &Arc<Self> is apparently possible (as of Rust 1.41) but using that with "async fn" means the returned future is tied to its lifetime. The workaround is to use async blocks as described here: <https://rust-lang.github.io/async-book/03_async_await/01_chapter.html> but that's really ugly: it brings back the explicit Future reference, requires futures::future::Either in some cases, and introduces another level of indenting. I think it's better to just pay the arc costs which are probably negligible, or at least cheaper than the boxing was before. Oh, and I make this compile on Rust 1.40 again as it claimed to. http-serve accidentally used the &Arc<Self> thing which broke this. Update to a freshly-pushed commit which doesn't do that.
2020-05-30 19:08:54 -07:00
let mut builder = mp4::FileBuilder::new(mp4::Type::MediaSegment);
pass prev duration and runs through API layer Builds on f3ddbfe, for #32 and #59.
2020-06-09 22:06:03 -07:00
let mut row = None;
more readable async web code This uses "async fn" throughout rather than a mix of async and the older futures style. And it takes advantage of the "self: Arc<Self>" syntax to avoid having a ServiceInner. It was confusing to have some methods on Service and some on ServiceInner; now that distinction is gone. One downside is there's a little more atomic reference-counting. Before, service_fn essentially took an &Arc<Self>, which means it could call Arc::clone where its use of self actually outlived the future (see stream_live_m4s) but didn't need to otherwise. After, it calls an async fn that takes Arc<Self>. Using &Arc<Self> is apparently possible (as of Rust 1.41) but using that with "async fn" means the returned future is tied to its lifetime. The workaround is to use async blocks as described here: <https://rust-lang.github.io/async-book/03_async_await/01_chapter.html> but that's really ugly: it brings back the explicit Future reference, requires futures::future::Either in some cases, and introduces another level of indenting. I think it's better to just pay the arc costs which are probably negligible, or at least cheaper than the boxing was before. Oh, and I make this compile on Rust 1.40 again as it claimed to. http-serve accidentally used the &Arc<Self> thing which broke this. Update to a freshly-pushed commit which doesn't do that.
2020-05-30 19:08:54 -07:00
{
let db = self.db.lock();
let mut rows = 0;
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
db.list_recordings_by_id(stream_id, live.recording..live.recording + 1, &mut |r| {
more readable async web code This uses "async fn" throughout rather than a mix of async and the older futures style. And it takes advantage of the "self: Arc<Self>" syntax to avoid having a ServiceInner. It was confusing to have some methods on Service and some on ServiceInner; now that distinction is gone. One downside is there's a little more atomic reference-counting. Before, service_fn essentially took an &Arc<Self>, which means it could call Arc::clone where its use of self actually outlived the future (see stream_live_m4s) but didn't need to otherwise. After, it calls an async fn that takes Arc<Self>. Using &Arc<Self> is apparently possible (as of Rust 1.41) but using that with "async fn" means the returned future is tied to its lifetime. The workaround is to use async blocks as described here: <https://rust-lang.github.io/async-book/03_async_await/01_chapter.html> but that's really ugly: it brings back the explicit Future reference, requires futures::future::Either in some cases, and introduces another level of indenting. I think it's better to just pay the arc costs which are probably negligible, or at least cheaper than the boxing was before. Oh, and I make this compile on Rust 1.40 again as it claimed to. http-serve accidentally used the &Arc<Self> thing which broke this. Update to a freshly-pushed commit which doesn't do that.
2020-05-30 19:08:54 -07:00
rows += 1;
pass prev duration and runs through API layer Builds on f3ddbfe, for #32 and #59.
2020-06-09 22:06:03 -07:00
row = Some(r);
live stream frame-by-frame rather than GOP-by-GOP (#59) This should reduce live stream latency by two seconds when my cameras are at their default setting (I frame interval = 2 * frame rate)! I was under the impression that every HTML5 Media Source Extensions media segment had to start with a Random Access Point. This used to be true, but apparently changed quite a while ago: https://bugs.chromium.org/p/chromium/issues/detail?id=229412 Support generating segments that don't start with a key frame, and plumb this through the mp4 media segment generation logic. Add some extra error checking in mp4 slice handling, as my first attempts had a mismatch between expected and actual lengths that silently returned corrupted .m4s files. Also pull everything from the most recent key frame on along with the first live segment to reduce startup latency. Live view is quite a bit more pleasant now.
2020-08-07 15:30:22 -07:00
builder.append(&db, r, live.media_off_90k.clone(), start_at_key)?;
more readable async web code This uses "async fn" throughout rather than a mix of async and the older futures style. And it takes advantage of the "self: Arc<Self>" syntax to avoid having a ServiceInner. It was confusing to have some methods on Service and some on ServiceInner; now that distinction is gone. One downside is there's a little more atomic reference-counting. Before, service_fn essentially took an &Arc<Self>, which means it could call Arc::clone where its use of self actually outlived the future (see stream_live_m4s) but didn't need to otherwise. After, it calls an async fn that takes Arc<Self>. Using &Arc<Self> is apparently possible (as of Rust 1.41) but using that with "async fn" means the returned future is tied to its lifetime. The workaround is to use async blocks as described here: <https://rust-lang.github.io/async-book/03_async_await/01_chapter.html> but that's really ugly: it brings back the explicit Future reference, requires futures::future::Either in some cases, and introduces another level of indenting. I think it's better to just pay the arc costs which are probably negligible, or at least cheaper than the boxing was before. Oh, and I make this compile on Rust 1.40 again as it claimed to. http-serve accidentally used the &Arc<Self> thing which broke this. Update to a freshly-pushed commit which doesn't do that.
2020-05-30 19:08:54 -07:00
Ok(())
})?;
if rows != 1 {
bail_t!(Internal, "unable to find {:?}", live);
}
}
pass prev duration and runs through API layer Builds on f3ddbfe, for #32 and #59.
2020-06-09 22:06:03 -07:00
let row = row.unwrap();
more readable async web code This uses "async fn" throughout rather than a mix of async and the older futures style. And it takes advantage of the "self: Arc<Self>" syntax to avoid having a ServiceInner. It was confusing to have some methods on Service and some on ServiceInner; now that distinction is gone. One downside is there's a little more atomic reference-counting. Before, service_fn essentially took an &Arc<Self>, which means it could call Arc::clone where its use of self actually outlived the future (see stream_live_m4s) but didn't need to otherwise. After, it calls an async fn that takes Arc<Self>. Using &Arc<Self> is apparently possible (as of Rust 1.41) but using that with "async fn" means the returned future is tied to its lifetime. The workaround is to use async blocks as described here: <https://rust-lang.github.io/async-book/03_async_await/01_chapter.html> but that's really ugly: it brings back the explicit Future reference, requires futures::future::Either in some cases, and introduces another level of indenting. I think it's better to just pay the arc costs which are probably negligible, or at least cheaper than the boxing was before. Oh, and I make this compile on Rust 1.40 again as it claimed to. http-serve accidentally used the &Arc<Self> thing which broke this. Update to a freshly-pushed commit which doesn't do that.
2020-05-30 19:08:54 -07:00
use http_serve::Entity;
let mp4 = builder.build(self.db.clone(), self.dirs_by_stream_id.clone())?;
let mut hdrs = header::HeaderMap::new();
mp4.add_headers(&mut hdrs);
let mime_type = hdrs.get(header::CONTENT_TYPE).unwrap();
start splitting wall and media duration for #34 This splits the schema and playback path. The recording path still adjusts the frame durations and always says the wall and media durations are the same. I expect to change that in a following commit. I wouldn't be surprised if that shakes out some bugs in this portion.
2020-08-04 21:44:01 -07:00
let (prev_media_duration, prev_runs) = row.prev_media_duration_and_runs.unwrap();
more readable async web code This uses "async fn" throughout rather than a mix of async and the older futures style. And it takes advantage of the "self: Arc<Self>" syntax to avoid having a ServiceInner. It was confusing to have some methods on Service and some on ServiceInner; now that distinction is gone. One downside is there's a little more atomic reference-counting. Before, service_fn essentially took an &Arc<Self>, which means it could call Arc::clone where its use of self actually outlived the future (see stream_live_m4s) but didn't need to otherwise. After, it calls an async fn that takes Arc<Self>. Using &Arc<Self> is apparently possible (as of Rust 1.41) but using that with "async fn" means the returned future is tied to its lifetime. The workaround is to use async blocks as described here: <https://rust-lang.github.io/async-book/03_async_await/01_chapter.html> but that's really ugly: it brings back the explicit Future reference, requires futures::future::Either in some cases, and introduces another level of indenting. I think it's better to just pay the arc costs which are probably negligible, or at least cheaper than the boxing was before. Oh, and I make this compile on Rust 1.40 again as it claimed to. http-serve accidentally used the &Arc<Self> thing which broke this. Update to a freshly-pushed commit which doesn't do that.
2020-05-30 19:08:54 -07:00
let hdr = format!(
"Content-Type: {}\r\n\
X-Recording-Start: {}\r\n\
X-Recording-Id: {}.{}\r\n\
fix live view This broke with the media vs wall duration split, part of #34.
2020-08-07 10:16:06 -07:00
X-Media-Time-Range: {}-{}\r\n\
start splitting wall and media duration for #34 This splits the schema and playback path. The recording path still adjusts the frame durations and always says the wall and media durations are the same. I expect to change that in a following commit. I wouldn't be surprised if that shakes out some bugs in this portion.
2020-08-04 21:44:01 -07:00
X-Prev-Media-Duration: {}\r\n\
pass prev duration and runs through API layer Builds on f3ddbfe, for #32 and #59.
2020-06-09 22:06:03 -07:00
X-Runs: {}\r\n\
Merge branch 'master' into new-schema
2020-06-02 22:58:11 -07:00
X-Video-Sample-Entry-Id: {}\r\n\r\n",
more readable async web code This uses "async fn" throughout rather than a mix of async and the older futures style. And it takes advantage of the "self: Arc<Self>" syntax to avoid having a ServiceInner. It was confusing to have some methods on Service and some on ServiceInner; now that distinction is gone. One downside is there's a little more atomic reference-counting. Before, service_fn essentially took an &Arc<Self>, which means it could call Arc::clone where its use of self actually outlived the future (see stream_live_m4s) but didn't need to otherwise. After, it calls an async fn that takes Arc<Self>. Using &Arc<Self> is apparently possible (as of Rust 1.41) but using that with "async fn" means the returned future is tied to its lifetime. The workaround is to use async blocks as described here: <https://rust-lang.github.io/async-book/03_async_await/01_chapter.html> but that's really ugly: it brings back the explicit Future reference, requires futures::future::Either in some cases, and introduces another level of indenting. I think it's better to just pay the arc costs which are probably negligible, or at least cheaper than the boxing was before. Oh, and I make this compile on Rust 1.40 again as it claimed to. http-serve accidentally used the &Arc<Self> thing which broke this. Update to a freshly-pushed commit which doesn't do that.
2020-05-30 19:08:54 -07:00
mime_type.to_str().unwrap(),
pass prev duration and runs through API layer Builds on f3ddbfe, for #32 and #59.
2020-06-09 22:06:03 -07:00
row.start.0,
more readable async web code This uses "async fn" throughout rather than a mix of async and the older futures style. And it takes advantage of the "self: Arc<Self>" syntax to avoid having a ServiceInner. It was confusing to have some methods on Service and some on ServiceInner; now that distinction is gone. One downside is there's a little more atomic reference-counting. Before, service_fn essentially took an &Arc<Self>, which means it could call Arc::clone where its use of self actually outlived the future (see stream_live_m4s) but didn't need to otherwise. After, it calls an async fn that takes Arc<Self>. Using &Arc<Self> is apparently possible (as of Rust 1.41) but using that with "async fn" means the returned future is tied to its lifetime. The workaround is to use async blocks as described here: <https://rust-lang.github.io/async-book/03_async_await/01_chapter.html> but that's really ugly: it brings back the explicit Future reference, requires futures::future::Either in some cases, and introduces another level of indenting. I think it's better to just pay the arc costs which are probably negligible, or at least cheaper than the boxing was before. Oh, and I make this compile on Rust 1.40 again as it claimed to. http-serve accidentally used the &Arc<Self> thing which broke this. Update to a freshly-pushed commit which doesn't do that.
2020-05-30 19:08:54 -07:00
open_id,
live.recording,
fix live view This broke with the media vs wall duration split, part of #34.
2020-08-07 10:16:06 -07:00
live.media_off_90k.start,
live.media_off_90k.end,
start splitting wall and media duration for #34 This splits the schema and playback path. The recording path still adjusts the frame durations and always says the wall and media durations are the same. I expect to change that in a following commit. I wouldn't be surprised if that shakes out some bugs in this portion.
2020-08-04 21:44:01 -07:00
prev_media_duration.0,
pass prev duration and runs through API layer Builds on f3ddbfe, for #32 and #59.
2020-06-09 22:06:03 -07:00
prev_runs + if row.run_offset == 0 { 1 } else { 0 },
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
&row.video_sample_entry_id
);
pass prev duration and runs through API layer Builds on f3ddbfe, for #32 and #59.
2020-06-09 22:06:03 -07:00
let mut v = hdr.into_bytes();
more readable async web code This uses "async fn" throughout rather than a mix of async and the older futures style. And it takes advantage of the "self: Arc<Self>" syntax to avoid having a ServiceInner. It was confusing to have some methods on Service and some on ServiceInner; now that distinction is gone. One downside is there's a little more atomic reference-counting. Before, service_fn essentially took an &Arc<Self>, which means it could call Arc::clone where its use of self actually outlived the future (see stream_live_m4s) but didn't need to otherwise. After, it calls an async fn that takes Arc<Self>. Using &Arc<Self> is apparently possible (as of Rust 1.41) but using that with "async fn" means the returned future is tied to its lifetime. The workaround is to use async blocks as described here: <https://rust-lang.github.io/async-book/03_async_await/01_chapter.html> but that's really ugly: it brings back the explicit Future reference, requires futures::future::Either in some cases, and introduces another level of indenting. I think it's better to just pay the arc costs which are probably negligible, or at least cheaper than the boxing was before. Oh, and I make this compile on Rust 1.40 again as it claimed to. http-serve accidentally used the &Arc<Self> thing which broke this. Update to a freshly-pushed commit which doesn't do that.
2020-05-30 19:08:54 -07:00
mp4.append_into_vec(&mut v).await?;
ws.send(tungstenite::Message::Binary(v)).await?;
Ok(())
}
async fn signals(&self, req: Request<hyper::Body>, caller: Caller) -> ResponseResult {
match *req.method() {
Method::POST => self.post_signals(req, caller).await,
Method::GET | Method::HEAD => self.get_signals(&req),
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
_ => Err(plain_response(
StatusCode::METHOD_NOT_ALLOWED,
"POST, GET, or HEAD expected",
small readability improvements to web.rs I think this has a minor behavior change: permission denied replies change to HTTP 403 where they were HTTP 401. The new behavior seems more correct, as these errors can occur when authentication has succeeded but the session in question is not authorized for the given operation. The UI currently doesn't care about this distinction.
2021-03-10 16:18:47 -08:00
)
.into()),
more readable async web code This uses "async fn" throughout rather than a mix of async and the older futures style. And it takes advantage of the "self: Arc<Self>" syntax to avoid having a ServiceInner. It was confusing to have some methods on Service and some on ServiceInner; now that distinction is gone. One downside is there's a little more atomic reference-counting. Before, service_fn essentially took an &Arc<Self>, which means it could call Arc::clone where its use of self actually outlived the future (see stream_live_m4s) but didn't need to otherwise. After, it calls an async fn that takes Arc<Self>. Using &Arc<Self> is apparently possible (as of Rust 1.41) but using that with "async fn" means the returned future is tied to its lifetime. The workaround is to use async blocks as described here: <https://rust-lang.github.io/async-book/03_async_await/01_chapter.html> but that's really ugly: it brings back the explicit Future reference, requires futures::future::Either in some cases, and introduces another level of indenting. I think it's better to just pay the arc costs which are probably negligible, or at least cheaper than the boxing was before. Oh, and I make this compile on Rust 1.40 again as it claimed to. http-serve accidentally used the &Arc<Self> thing which broke this. Update to a freshly-pushed commit which doesn't do that.
2020-05-30 19:08:54 -07:00
}
}
small readability improvements to web.rs I think this has a minor behavior change: permission denied replies change to HTTP 403 where they were HTTP 401. The new behavior seems more correct, as these errors can occur when authentication has succeeded but the session in question is not authorized for the given operation. The UI currently doesn't care about this distinction.
2021-03-10 16:18:47 -08:00
/// Serves an HTTP request.
/// Note that the `serve` wrapper handles responses the same whether they
/// are `Ok` or `Err`. But returning `Err` here with the `?` operator is
/// convenient for error paths.
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
async fn serve_inner(
self: Arc<Self>,
req: Request<::hyper::Body>,
p: Path,
caller: Caller,
) -> ResponseResult {
more readable async web code This uses "async fn" throughout rather than a mix of async and the older futures style. And it takes advantage of the "self: Arc<Self>" syntax to avoid having a ServiceInner. It was confusing to have some methods on Service and some on ServiceInner; now that distinction is gone. One downside is there's a little more atomic reference-counting. Before, service_fn essentially took an &Arc<Self>, which means it could call Arc::clone where its use of self actually outlived the future (see stream_live_m4s) but didn't need to otherwise. After, it calls an async fn that takes Arc<Self>. Using &Arc<Self> is apparently possible (as of Rust 1.41) but using that with "async fn" means the returned future is tied to its lifetime. The workaround is to use async blocks as described here: <https://rust-lang.github.io/async-book/03_async_await/01_chapter.html> but that's really ugly: it brings back the explicit Future reference, requires futures::future::Either in some cases, and introduces another level of indenting. I think it's better to just pay the arc costs which are probably negligible, or at least cheaper than the boxing was before. Oh, and I make this compile on Rust 1.40 again as it claimed to. http-serve accidentally used the &Arc<Self> thing which broke this. Update to a freshly-pushed commit which doesn't do that.
2020-05-30 19:08:54 -07:00
let (cache, mut response) = match p {
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
Path::InitSegment(sha1, debug) => (
CacheControl::PrivateStatic,
self.init_segment(sha1, debug, &req)?,
),
more readable async web code This uses "async fn" throughout rather than a mix of async and the older futures style. And it takes advantage of the "self: Arc<Self>" syntax to avoid having a ServiceInner. It was confusing to have some methods on Service and some on ServiceInner; now that distinction is gone. One downside is there's a little more atomic reference-counting. Before, service_fn essentially took an &Arc<Self>, which means it could call Arc::clone where its use of self actually outlived the future (see stream_live_m4s) but didn't need to otherwise. After, it calls an async fn that takes Arc<Self>. Using &Arc<Self> is apparently possible (as of Rust 1.41) but using that with "async fn" means the returned future is tied to its lifetime. The workaround is to use async blocks as described here: <https://rust-lang.github.io/async-book/03_async_await/01_chapter.html> but that's really ugly: it brings back the explicit Future reference, requires futures::future::Either in some cases, and introduces another level of indenting. I think it's better to just pay the arc costs which are probably negligible, or at least cheaper than the boxing was before. Oh, and I make this compile on Rust 1.40 again as it claimed to. http-serve accidentally used the &Arc<Self> thing which broke this. Update to a freshly-pushed commit which doesn't do that.
2020-05-30 19:08:54 -07:00
Path::TopLevel => (CacheControl::PrivateDynamic, self.top_level(&req, caller)?),
Path::Request => (CacheControl::PrivateDynamic, self.request(&req)?),
Path::Camera(uuid) => (CacheControl::PrivateDynamic, self.camera(&req, uuid)?),
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
Path::StreamRecordings(uuid, type_) => (
CacheControl::PrivateDynamic,
self.stream_recordings(&req, uuid, type_)?,
),
Path::StreamViewMp4(uuid, type_, debug) => (
CacheControl::PrivateStatic,
self.stream_view_mp4(&req, caller, uuid, type_, mp4::Type::Normal, debug)?,
),
Path::StreamViewMp4Segment(uuid, type_, debug) => (
CacheControl::PrivateStatic,
self.stream_view_mp4(&req, caller, uuid, type_, mp4::Type::MediaSegment, debug)?,
),
Path::StreamLiveMp4Segments(uuid, type_) => (
CacheControl::PrivateDynamic,
self.stream_live_m4s(req, caller, uuid, type_)?,
),
more readable async web code This uses "async fn" throughout rather than a mix of async and the older futures style. And it takes advantage of the "self: Arc<Self>" syntax to avoid having a ServiceInner. It was confusing to have some methods on Service and some on ServiceInner; now that distinction is gone. One downside is there's a little more atomic reference-counting. Before, service_fn essentially took an &Arc<Self>, which means it could call Arc::clone where its use of self actually outlived the future (see stream_live_m4s) but didn't need to otherwise. After, it calls an async fn that takes Arc<Self>. Using &Arc<Self> is apparently possible (as of Rust 1.41) but using that with "async fn" means the returned future is tied to its lifetime. The workaround is to use async blocks as described here: <https://rust-lang.github.io/async-book/03_async_await/01_chapter.html> but that's really ugly: it brings back the explicit Future reference, requires futures::future::Either in some cases, and introduces another level of indenting. I think it's better to just pay the arc costs which are probably negligible, or at least cheaper than the boxing was before. Oh, and I make this compile on Rust 1.40 again as it claimed to. http-serve accidentally used the &Arc<Self> thing which broke this. Update to a freshly-pushed commit which doesn't do that.
2020-05-30 19:08:54 -07:00
Path::NotFound => return Err(not_found("path not understood")),
Path::Login => (CacheControl::PrivateDynamic, self.login(req).await?),
Path::Logout => (CacheControl::PrivateDynamic, self.logout(req).await?),
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
Path::Signals => (
CacheControl::PrivateDynamic,
self.signals(req, caller).await?,
),
Path::Static => (CacheControl::None, self.static_file(req).await?),
UI preferences: #153 #155
2021-09-01 15:01:42 -07:00
Path::User(id) => (
CacheControl::PrivateDynamic,
self.user(req, caller, id).await?,
),
more readable async web code This uses "async fn" throughout rather than a mix of async and the older futures style. And it takes advantage of the "self: Arc<Self>" syntax to avoid having a ServiceInner. It was confusing to have some methods on Service and some on ServiceInner; now that distinction is gone. One downside is there's a little more atomic reference-counting. Before, service_fn essentially took an &Arc<Self>, which means it could call Arc::clone where its use of self actually outlived the future (see stream_live_m4s) but didn't need to otherwise. After, it calls an async fn that takes Arc<Self>. Using &Arc<Self> is apparently possible (as of Rust 1.41) but using that with "async fn" means the returned future is tied to its lifetime. The workaround is to use async blocks as described here: <https://rust-lang.github.io/async-book/03_async_await/01_chapter.html> but that's really ugly: it brings back the explicit Future reference, requires futures::future::Either in some cases, and introduces another level of indenting. I think it's better to just pay the arc costs which are probably negligible, or at least cheaper than the boxing was before. Oh, and I make this compile on Rust 1.40 again as it claimed to. http-serve accidentally used the &Arc<Self> thing which broke this. Update to a freshly-pushed commit which doesn't do that.
2020-05-30 19:08:54 -07:00
};
match cache {
CacheControl::PrivateStatic => {
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
response.headers_mut().insert(
header::CACHE_CONTROL,
HeaderValue::from_static("private, max-age=3600"),
);
}
more readable async web code This uses "async fn" throughout rather than a mix of async and the older futures style. And it takes advantage of the "self: Arc<Self>" syntax to avoid having a ServiceInner. It was confusing to have some methods on Service and some on ServiceInner; now that distinction is gone. One downside is there's a little more atomic reference-counting. Before, service_fn essentially took an &Arc<Self>, which means it could call Arc::clone where its use of self actually outlived the future (see stream_live_m4s) but didn't need to otherwise. After, it calls an async fn that takes Arc<Self>. Using &Arc<Self> is apparently possible (as of Rust 1.41) but using that with "async fn" means the returned future is tied to its lifetime. The workaround is to use async blocks as described here: <https://rust-lang.github.io/async-book/03_async_await/01_chapter.html> but that's really ugly: it brings back the explicit Future reference, requires futures::future::Either in some cases, and introduces another level of indenting. I think it's better to just pay the arc costs which are probably negligible, or at least cheaper than the boxing was before. Oh, and I make this compile on Rust 1.40 again as it claimed to. http-serve accidentally used the &Arc<Self> thing which broke this. Update to a freshly-pushed commit which doesn't do that.
2020-05-30 19:08:54 -07:00
CacheControl::PrivateDynamic => {
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
response.headers_mut().insert(
header::CACHE_CONTROL,
HeaderValue::from_static("private, no-cache"),
);
}
CacheControl::None => {}
more readable async web code This uses "async fn" throughout rather than a mix of async and the older futures style. And it takes advantage of the "self: Arc<Self>" syntax to avoid having a ServiceInner. It was confusing to have some methods on Service and some on ServiceInner; now that distinction is gone. One downside is there's a little more atomic reference-counting. Before, service_fn essentially took an &Arc<Self>, which means it could call Arc::clone where its use of self actually outlived the future (see stream_live_m4s) but didn't need to otherwise. After, it calls an async fn that takes Arc<Self>. Using &Arc<Self> is apparently possible (as of Rust 1.41) but using that with "async fn" means the returned future is tied to its lifetime. The workaround is to use async blocks as described here: <https://rust-lang.github.io/async-book/03_async_await/01_chapter.html> but that's really ugly: it brings back the explicit Future reference, requires futures::future::Either in some cases, and introduces another level of indenting. I think it's better to just pay the arc costs which are probably negligible, or at least cheaper than the boxing was before. Oh, and I make this compile on Rust 1.40 again as it claimed to. http-serve accidentally used the &Arc<Self> thing which broke this. Update to a freshly-pushed commit which doesn't do that.
2020-05-30 19:08:54 -07:00
}
Ok(response)
}
small readability improvements to web.rs I think this has a minor behavior change: permission denied replies change to HTTP 403 where they were HTTP 401. The new behavior seems more correct, as these errors can occur when authentication has succeeded but the session in question is not authorized for the given operation. The UI currently doesn't care about this distinction.
2021-03-10 16:18:47 -08:00
/// Serves an HTTP request.
/// An error return from this method causes hyper to abruptly drop the
/// HTTP connection rather than respond. That's not terribly useful, so this
/// method always returns `Ok`. It delegates to a `serve_inner` which is
/// allowed to generate `Err` results with the `?` operator, but returns
/// them to hyper as `Ok` results.
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
pub async fn serve(
self: Arc<Self>,
req: Request<::hyper::Body>,
) -> Result<Response<Body>, std::convert::Infallible> {
more readable async web code This uses "async fn" throughout rather than a mix of async and the older futures style. And it takes advantage of the "self: Arc<Self>" syntax to avoid having a ServiceInner. It was confusing to have some methods on Service and some on ServiceInner; now that distinction is gone. One downside is there's a little more atomic reference-counting. Before, service_fn essentially took an &Arc<Self>, which means it could call Arc::clone where its use of self actually outlived the future (see stream_live_m4s) but didn't need to otherwise. After, it calls an async fn that takes Arc<Self>. Using &Arc<Self> is apparently possible (as of Rust 1.41) but using that with "async fn" means the returned future is tied to its lifetime. The workaround is to use async blocks as described here: <https://rust-lang.github.io/async-book/03_async_await/01_chapter.html> but that's really ugly: it brings back the explicit Future reference, requires futures::future::Either in some cases, and introduces another level of indenting. I think it's better to just pay the arc costs which are probably negligible, or at least cheaper than the boxing was before. Oh, and I make this compile on Rust 1.40 again as it claimed to. http-serve accidentally used the &Arc<Self> thing which broke this. Update to a freshly-pushed commit which doesn't do that.
2020-05-30 19:08:54 -07:00
let p = Path::decode(req.uri().path());
address some no-op clippy warnings
2021-05-17 14:31:50 -07:00
let always_allow_unauthenticated = matches!(
p,
Path::NotFound | Path::Request | Path::Login | Path::Logout | Path::Static
);
more readable async web code This uses "async fn" throughout rather than a mix of async and the older futures style. And it takes advantage of the "self: Arc<Self>" syntax to avoid having a ServiceInner. It was confusing to have some methods on Service and some on ServiceInner; now that distinction is gone. One downside is there's a little more atomic reference-counting. Before, service_fn essentially took an &Arc<Self>, which means it could call Arc::clone where its use of self actually outlived the future (see stream_live_m4s) but didn't need to otherwise. After, it calls an async fn that takes Arc<Self>. Using &Arc<Self> is apparently possible (as of Rust 1.41) but using that with "async fn" means the returned future is tied to its lifetime. The workaround is to use async blocks as described here: <https://rust-lang.github.io/async-book/03_async_await/01_chapter.html> but that's really ugly: it brings back the explicit Future reference, requires futures::future::Either in some cases, and introduces another level of indenting. I think it's better to just pay the arc costs which are probably negligible, or at least cheaper than the boxing was before. Oh, and I make this compile on Rust 1.40 again as it claimed to. http-serve accidentally used the &Arc<Self> thing which broke this. Update to a freshly-pushed commit which doesn't do that.
2020-05-30 19:08:54 -07:00
debug!("request on: {}: {:?}", req.uri(), p);
let caller = match self.authenticate(&req, always_allow_unauthenticated) {
Ok(c) => c,
Err(e) => return Ok(from_base_error(e)),
};
small readability improvements to web.rs I think this has a minor behavior change: permission denied replies change to HTTP 403 where they were HTTP 401. The new behavior seems more correct, as these errors can occur when authentication has succeeded but the session in question is not authorized for the given operation. The UI currently doesn't care about this distinction.
2021-03-10 16:18:47 -08:00
Ok(self
.serve_inner(req, p, caller)
.await
.unwrap_or_else(|e| e.0))
more readable async web code This uses "async fn" throughout rather than a mix of async and the older futures style. And it takes advantage of the "self: Arc<Self>" syntax to avoid having a ServiceInner. It was confusing to have some methods on Service and some on ServiceInner; now that distinction is gone. One downside is there's a little more atomic reference-counting. Before, service_fn essentially took an &Arc<Self>, which means it could call Arc::clone where its use of self actually outlived the future (see stream_live_m4s) but didn't need to otherwise. After, it calls an async fn that takes Arc<Self>. Using &Arc<Self> is apparently possible (as of Rust 1.41) but using that with "async fn" means the returned future is tied to its lifetime. The workaround is to use async blocks as described here: <https://rust-lang.github.io/async-book/03_async_await/01_chapter.html> but that's really ugly: it brings back the explicit Future reference, requires futures::future::Either in some cases, and introduces another level of indenting. I think it's better to just pay the arc costs which are probably negligible, or at least cheaper than the boxing was before. Oh, and I make this compile on Rust 1.40 again as it claimed to. http-serve accidentally used the &Arc<Self> thing which broke this. Update to a freshly-pushed commit which doesn't do that.
2020-05-30 19:08:54 -07:00
}
add concept of user/session permissions (I also considered the names "capabilities" and "scopes", but I think "permissions" is the most widely understood.) This is increasingly necessary as the web API becomes more capable. Among other things, it allows: * non-administrator users who can view but not access camera passwords or change any state * workers that update signal state based on cameras' built-in motion detection or a security system's events but don't need to view videos * control over what can be done without authenticating Currently session permissions are just copied from user permissions, but you can also imagine admin sessions vs not, as a checkbox when signing in. This would match the standard Unix workflow of using a non-administrative session most of the time. Relevant to my current signals work (#28) and to the addition of an administrative API (#35, including #66).
2019-06-19 15:17:50 -07:00
fn top_level(&self, req: &Request<::hyper::Body>, caller: Caller) -> ResponseResult {
add a basic Javascript UI The Javascript is pretty amateurish I'm sure but at least it's something to iterate from. It's already much more pleasant for browsing through videos in several ways: * more responsive to load only a day at a time rather than 90+ days * much easier to see the same time segment on several cameras * more pleasant to have the videos load as a popup rather than a link that blows away your position in an enormous list * exposes the fancier .mp4 generation options: splitting at lengths other than the default, trimming to an arbitrary start and end time, including a subtitle track with timestamps. There's a slight regression in functionality: I didn't match the former top-level page which showed how much camera used of its disk allocation and the total duration of video. This is exposed in the JSON API, so it shouldn't be too hard to add back.
2017-10-21 21:54:27 -07:00
let mut days = false;
add concept of user/session permissions (I also considered the names "capabilities" and "scopes", but I think "permissions" is the most widely understood.) This is increasingly necessary as the web API becomes more capable. Among other things, it allows: * non-administrator users who can view but not access camera passwords or change any state * workers that update signal state based on cameras' built-in motion detection or a security system's events but don't need to view videos * control over what can be done without authenticating Currently session permissions are just copied from user permissions, but you can also imagine admin sessions vs not, as a checkbox when signing in. This would match the standard Unix workflow of using a non-administrative session most of the time. Relevant to my current signals work (#28) and to the addition of an administrative API (#35, including #66).
2019-06-19 15:17:50 -07:00
let mut camera_configs = false;
gzip json responses, handle HEAD properly
2018-01-23 11:22:23 -08:00
if let Some(q) = req.uri().query() {
add a basic Javascript UI The Javascript is pretty amateurish I'm sure but at least it's something to iterate from. It's already much more pleasant for browsing through videos in several ways: * more responsive to load only a day at a time rather than 90+ days * much easier to see the same time segment on several cameras * more pleasant to have the videos load as a popup rather than a link that blows away your position in an enormous list * exposes the fancier .mp4 generation options: splitting at lengths other than the default, trimming to an arbitrary start and end time, including a subtitle track with timestamps. There's a slight regression in functionality: I didn't match the former top-level page which showed how much camera used of its disk allocation and the total duration of video. This is exposed in the JSON API, so it shouldn't be too hard to add back.
2017-10-21 21:54:27 -07:00
for (key, value) in form_urlencoded::parse(q.as_bytes()) {
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
let (key, value): (_, &str) = (key.borrow(), value.borrow());
add a basic Javascript UI The Javascript is pretty amateurish I'm sure but at least it's something to iterate from. It's already much more pleasant for browsing through videos in several ways: * more responsive to load only a day at a time rather than 90+ days * much easier to see the same time segment on several cameras * more pleasant to have the videos load as a popup rather than a link that blows away your position in an enormous list * exposes the fancier .mp4 generation options: splitting at lengths other than the default, trimming to an arbitrary start and end time, including a subtitle track with timestamps. There's a slight regression in functionality: I didn't match the former top-level page which showed how much camera used of its disk allocation and the total duration of video. This is exposed in the JSON API, so it shouldn't be too hard to add back.
2017-10-21 21:54:27 -07:00
match key {
"days" => days = value == "true",
add concept of user/session permissions (I also considered the names "capabilities" and "scopes", but I think "permissions" is the most widely understood.) This is increasingly necessary as the web API becomes more capable. Among other things, it allows: * non-administrator users who can view but not access camera passwords or change any state * workers that update signal state based on cameras' built-in motion detection or a security system's events but don't need to view videos * control over what can be done without authenticating Currently session permissions are just copied from user permissions, but you can also imagine admin sessions vs not, as a checkbox when signing in. This would match the standard Unix workflow of using a non-administrative session most of the time. Relevant to my current signals work (#28) and to the addition of an administrative API (#35, including #66).
2019-06-19 15:17:50 -07:00
"cameraConfigs" => camera_configs = value == "true",
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
_ => {}
add a basic Javascript UI The Javascript is pretty amateurish I'm sure but at least it's something to iterate from. It's already much more pleasant for browsing through videos in several ways: * more responsive to load only a day at a time rather than 90+ days * much easier to see the same time segment on several cameras * more pleasant to have the videos load as a popup rather than a link that blows away your position in an enormous list * exposes the fancier .mp4 generation options: splitting at lengths other than the default, trimming to an arbitrary start and end time, including a subtitle track with timestamps. There's a slight regression in functionality: I didn't match the former top-level page which showed how much camera used of its disk allocation and the total duration of video. This is exposed in the JSON API, so it shouldn't be too hard to add back.
2017-10-21 21:54:27 -07:00
};
}
}
address some no-op clippy warnings
2021-05-17 14:31:50 -07:00
if camera_configs && !caller.permissions.read_camera_configs {
bail_t!(PermissionDenied, "read_camera_configs required");
add concept of user/session permissions (I also considered the names "capabilities" and "scopes", but I think "permissions" is the most widely understood.) This is increasingly necessary as the web API becomes more capable. Among other things, it allows: * non-administrator users who can view but not access camera passwords or change any state * workers that update signal state based on cameras' built-in motion detection or a security system's events but don't need to view videos * control over what can be done without authenticating Currently session permissions are just copied from user permissions, but you can also imagine admin sessions vs not, as a checkbox when signing in. This would match the standard Unix workflow of using a non-administrative session most of the time. Relevant to my current signals work (#28) and to the addition of an administrative API (#35, including #66).
2019-06-19 15:17:50 -07:00
}
read-only signals support (#28) This is mostly untested and useless by itself, but it's a starting point. In particular: * there's no way to set up signals or add/remove/update events yet except by manual changes to the database. * if you associate a signal with a camera then remove the camera, hitting /api/ will error out.
2019-06-06 16:18:13 -07:00
let db = self.db.lock();
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
serve_json(
req,
&json::TopLevel {
time_zone_name: &self.time_zone_name,
add server version to /api response Fixes #171
2021-10-27 13:09:20 -07:00
server_version: env!("CARGO_PKG_VERSION"),
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
cameras: (&db, days, camera_configs),
UI preferences: #153 #155
2021-09-01 15:01:42 -07:00
user: caller.user,
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
signals: (&db, days),
signal_types: &db,
},
)
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 14:34:00 -08:00
}
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
fn camera(&self, req: &Request<::hyper::Body>, uuid: Uuid) -> ResponseResult {
read-only signals support (#28) This is mostly untested and useless by itself, but it's a starting point. In particular: * there's no way to set up signals or add/remove/update events yet except by manual changes to the database. * if you associate a signal with a camera then remove the camera, hitting /api/ will error out.
2019-06-06 16:18:13 -07:00
let db = self.db.lock();
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
let camera = db
.get_camera(uuid)
.ok_or_else(|| not_found(format!("no such camera {}", uuid)))?;
serve_json(
req,
&json::Camera::wrap(camera, &db, true, false).map_err(internal_server_err)?,
)
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 14:34:00 -08:00
}
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
fn stream_recordings(
&self,
req: &Request<::hyper::Body>,
uuid: Uuid,
type_: db::StreamType,
) -> ResponseResult {
support run splitting in json api
2017-10-17 09:00:05 -07:00
let (r, split) = {
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
let mut time = recording::Time::min_value()..recording::Time::max_value();
support run splitting in json api
2017-10-17 09:00:05 -07:00
let mut split = recording::Duration(i64::max_value());
gzip json responses, handle HEAD properly
2018-01-23 11:22:23 -08:00
if let Some(q) = req.uri().query() {
support run splitting in json api
2017-10-17 09:00:05 -07:00
for (key, value) in form_urlencoded::parse(q.as_bytes()) {
let (key, value) = (key.borrow(), value.borrow());
match key {
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
"startTime90k" => {
time.start = recording::Time::parse(value)
.map_err(|_| bad_req("unparseable startTime90k"))?
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
}
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
"endTime90k" => {
time.end = recording::Time::parse(value)
.map_err(|_| bad_req("unparseable endTime90k"))?
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
}
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
"split90k" => {
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
split = recording::Duration(
i64::from_str(value)
.map_err(|_| bad_req("unparseable split90k"))?,
)
}
_ => {}
support run splitting in json api
2017-10-17 09:00:05 -07:00
}
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
}
support run splitting in json api
2017-10-17 09:00:05 -07:00
}
(time, split)
};
reorganize /recordings JSON response I want to start returning the pixel aspect ratio of each video sample entry. It's silly to duplicate it for each returned recording, so let's instead return a videoSampleEntryId and then put all the information about each VSE once. This change doesn't actually handle pixel aspect ratio server-side yet. Most likely I'll require a new schema version for that, to store it as a new column in the database. Codec-specific logic in the database layer is awkward and I'd like to avoid it. I did a similar schema change to add the rfc6381_codec. I also adjusted ui-src/lib/models/Recording.js in a few ways: * fixed a couple mismatches between its field name and the key defined in the API. Consistency aids understanding. * dropped all the getters in favor of just setting the fields (with type annotations) as described here: https://google.github.io/styleguide/jsguide.html#features-classes-fields * where the wire format used undefined (to save space), translate it to a more natural null or false.
2020-03-13 21:20:51 -07:00
let db = self.db.lock();
let mut out = json::ListRecordings {
recordings: Vec::new(),
video_sample_entries: (&db, Vec::new()),
};
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
let camera = db.get_camera(uuid).ok_or_else(|| {
plain_response(StatusCode::NOT_FOUND, format!("no such camera {}", uuid))
})?;
let stream_id = camera.streams[type_.index()].ok_or_else(|| {
plain_response(
StatusCode::NOT_FOUND,
format!("no such stream {}/{}", uuid, type_),
)
})?;
reorganize /recordings JSON response I want to start returning the pixel aspect ratio of each video sample entry. It's silly to duplicate it for each returned recording, so let's instead return a videoSampleEntryId and then put all the information about each VSE once. This change doesn't actually handle pixel aspect ratio server-side yet. Most likely I'll require a new schema version for that, to store it as a new column in the database. Codec-specific logic in the database layer is awkward and I'd like to avoid it. I did a similar schema change to add the rfc6381_codec. I also adjusted ui-src/lib/models/Recording.js in a few ways: * fixed a couple mismatches between its field name and the key defined in the API. Consistency aids understanding. * dropped all the getters in favor of just setting the fields (with type annotations) as described here: https://google.github.io/styleguide/jsguide.html#features-classes-fields * where the wire format used undefined (to save space), translate it to a more natural null or false.
2020-03-13 21:20:51 -07:00
db.list_aggregated_recordings(stream_id, r, split, &mut |row| {
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
let end = row.ids.end - 1; // in api, ids are inclusive.
reorganize /recordings JSON response I want to start returning the pixel aspect ratio of each video sample entry. It's silly to duplicate it for each returned recording, so let's instead return a videoSampleEntryId and then put all the information about each VSE once. This change doesn't actually handle pixel aspect ratio server-side yet. Most likely I'll require a new schema version for that, to store it as a new column in the database. Codec-specific logic in the database layer is awkward and I'd like to avoid it. I did a similar schema change to add the rfc6381_codec. I also adjusted ui-src/lib/models/Recording.js in a few ways: * fixed a couple mismatches between its field name and the key defined in the API. Consistency aids understanding. * dropped all the getters in favor of just setting the fields (with type annotations) as described here: https://google.github.io/styleguide/jsguide.html#features-classes-fields * where the wire format used undefined (to save space), translate it to a more natural null or false.
2020-03-13 21:20:51 -07:00
out.recordings.push(json::Recording {
start_id: row.ids.start,
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
end_id: if end == row.ids.start {
None
} else {
Some(end)
},
reorganize /recordings JSON response I want to start returning the pixel aspect ratio of each video sample entry. It's silly to duplicate it for each returned recording, so let's instead return a videoSampleEntryId and then put all the information about each VSE once. This change doesn't actually handle pixel aspect ratio server-side yet. Most likely I'll require a new schema version for that, to store it as a new column in the database. Codec-specific logic in the database layer is awkward and I'd like to avoid it. I did a similar schema change to add the rfc6381_codec. I also adjusted ui-src/lib/models/Recording.js in a few ways: * fixed a couple mismatches between its field name and the key defined in the API. Consistency aids understanding. * dropped all the getters in favor of just setting the fields (with type annotations) as described here: https://google.github.io/styleguide/jsguide.html#features-classes-fields * where the wire format used undefined (to save space), translate it to a more natural null or false.
2020-03-13 21:20:51 -07:00
start_time_90k: row.time.start.0,
end_time_90k: row.time.end.0,
sample_file_bytes: row.sample_file_bytes,
open_id: row.open_id,
first_uncommitted: row.first_uncommitted,
video_samples: row.video_samples,
minor api fixes * videoSampleEntryId should be a number, not a string * fix some markdown formatting * fix missing description of sampleFileBytes
2021-02-22 13:46:51 -08:00
video_sample_entry_id: row.video_sample_entry_id,
reorganize /recordings JSON response I want to start returning the pixel aspect ratio of each video sample entry. It's silly to duplicate it for each returned recording, so let's instead return a videoSampleEntryId and then put all the information about each VSE once. This change doesn't actually handle pixel aspect ratio server-side yet. Most likely I'll require a new schema version for that, to store it as a new column in the database. Codec-specific logic in the database layer is awkward and I'd like to avoid it. I did a similar schema change to add the rfc6381_codec. I also adjusted ui-src/lib/models/Recording.js in a few ways: * fixed a couple mismatches between its field name and the key defined in the API. Consistency aids understanding. * dropped all the getters in favor of just setting the fields (with type annotations) as described here: https://google.github.io/styleguide/jsguide.html#features-classes-fields * where the wire format used undefined (to save space), translate it to a more natural null or false.
2020-03-13 21:20:51 -07:00
growing: row.growing,
UNTESTED: note trailing zeros in /recordings reply
2021-10-10 16:13:57 -07:00
has_trailing_zero: row.has_trailing_zero,
reorganize /recordings JSON response I want to start returning the pixel aspect ratio of each video sample entry. It's silly to duplicate it for each returned recording, so let's instead return a videoSampleEntryId and then put all the information about each VSE once. This change doesn't actually handle pixel aspect ratio server-side yet. Most likely I'll require a new schema version for that, to store it as a new column in the database. Codec-specific logic in the database layer is awkward and I'd like to avoid it. I did a similar schema change to add the rfc6381_codec. I also adjusted ui-src/lib/models/Recording.js in a few ways: * fixed a couple mismatches between its field name and the key defined in the API. Consistency aids understanding. * dropped all the getters in favor of just setting the fields (with type annotations) as described here: https://google.github.io/styleguide/jsguide.html#features-classes-fields * where the wire format used undefined (to save space), translate it to a more natural null or false.
2020-03-13 21:20:51 -07:00
});
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
if !out
.video_sample_entries
.1
.contains(&row.video_sample_entry_id)
{
reorganize /recordings JSON response I want to start returning the pixel aspect ratio of each video sample entry. It's silly to duplicate it for each returned recording, so let's instead return a videoSampleEntryId and then put all the information about each VSE once. This change doesn't actually handle pixel aspect ratio server-side yet. Most likely I'll require a new schema version for that, to store it as a new column in the database. Codec-specific logic in the database layer is awkward and I'd like to avoid it. I did a similar schema change to add the rfc6381_codec. I also adjusted ui-src/lib/models/Recording.js in a few ways: * fixed a couple mismatches between its field name and the key defined in the API. Consistency aids understanding. * dropped all the getters in favor of just setting the fields (with type annotations) as described here: https://google.github.io/styleguide/jsguide.html#features-classes-fields * where the wire format used undefined (to save space), translate it to a more natural null or false.
2020-03-13 21:20:51 -07:00
out.video_sample_entries.1.push(row.video_sample_entry_id);
}
Ok(())
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
})
.map_err(internal_server_err)?;
read-only signals support (#28) This is mostly untested and useless by itself, but it's a starting point. In particular: * there's no way to set up signals or add/remove/update events yet except by manual changes to the database. * if you associate a signal with a camera then remove the camera, hitting /api/ will error out.
2019-06-06 16:18:13 -07:00
serve_json(req, &out)
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 14:34:00 -08:00
}
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
fn init_segment(&self, id: i32, debug: bool, req: &Request<::hyper::Body>) -> ResponseResult {
support segmented mp4s This is intended to support HTML5 Media Source Extensions, which I expect to be the most practical way to make a good web UI with a proper scrub bar and such. This feature has had very limited testing on Chrome and Firefox, and that was not entirely successful. More work is needed before it's usable, but this seems like a helpful progress checkpoint.
2017-10-01 15:29:22 -07:00
let mut builder = mp4::FileBuilder::new(mp4::Type::InitSegment);
let db = self.db.lock();
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
let ent = db
.video_sample_entries_by_id()
.get(&id)
use Blake3 instead of SHA-1 or Blake2b Benefits: * Blake3 is faster. This is most noticeable for the hashing of the sample file data. * we no longer need OpenSSL, which helps with shrinking the binary size (#70). sha1 basically forced OpenSSL usage; ring deliberately doesn't support this old algorithm, and the pure-Rust sha1 crate is painfully slow. OpenSSL might still be a better choice than ring/rustls for TLS but it's nice to have the option. For the video sample entries, I decided we don't need to hash at all. I think the id number is sufficiently stable, and it's okay---perhaps even desirable---if an existing init segment changes for fixes like e5b83c2.
2020-03-20 20:52:30 -07:00
.ok_or_else(|| not_found("not such init segment"))?;
builder.append_video_sample_entry(ent.clone());
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
let mp4 = builder
.build(self.db.clone(), self.dirs_by_stream_id.clone())
use Blake3 instead of SHA-1 or Blake2b Benefits: * Blake3 is faster. This is most noticeable for the hashing of the sample file data. * we no longer need OpenSSL, which helps with shrinking the binary size (#70). sha1 basically forced OpenSSL usage; ring deliberately doesn't support this old algorithm, and the pure-Rust sha1 crate is painfully slow. OpenSSL might still be a better choice than ring/rustls for TLS but it's nice to have the option. For the video sample entries, I decided we don't need to hash at all. I think the id number is sufficiently stable, and it's okay---perhaps even desirable---if an existing init segment changes for fixes like e5b83c2.
2020-03-20 20:52:30 -07:00
.map_err(from_base_error)?;
if debug {
Ok(plain_response(StatusCode::OK, format!("{:#?}", mp4)))
} else {
Ok(http_serve::serve(mp4, req))
support segmented mp4s This is intended to support HTML5 Media Source Extensions, which I expect to be the most practical way to make a good web UI with a proper scrub bar and such. This feature has had very limited testing on Chrome and Firefox, and that was not entirely successful. More work is needed before it's usable, but this seems like a helpful progress checkpoint.
2017-10-01 15:29:22 -07:00
}
}
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
fn stream_view_mp4(
&self,
req: &Request<::hyper::Body>,
caller: Caller,
uuid: Uuid,
stream_type: db::StreamType,
mp4_type: mp4::Type,
debug: bool,
) -> ResponseResult {
add concept of user/session permissions (I also considered the names "capabilities" and "scopes", but I think "permissions" is the most widely understood.) This is increasingly necessary as the web API becomes more capable. Among other things, it allows: * non-administrator users who can view but not access camera passwords or change any state * workers that update signal state based on cameras' built-in motion detection or a security system's events but don't need to view videos * control over what can be done without authenticating Currently session permissions are just copied from user permissions, but you can also imagine admin sessions vs not, as a checkbox when signing in. This would match the standard Unix workflow of using a non-administrative session most of the time. Relevant to my current signals work (#28) and to the addition of an administrative API (#35, including #66).
2019-06-19 15:17:50 -07:00
if !caller.permissions.view_video {
small readability improvements to web.rs I think this has a minor behavior change: permission denied replies change to HTTP 403 where they were HTTP 401. The new behavior seems more correct, as these errors can occur when authentication has succeeded but the session in question is not authorized for the given operation. The UI currently doesn't care about this distinction.
2021-03-10 16:18:47 -08:00
bail_t!(PermissionDenied, "view_video required");
add concept of user/session permissions (I also considered the names "capabilities" and "scopes", but I think "permissions" is the most widely understood.) This is increasingly necessary as the web API becomes more capable. Among other things, it allows: * non-administrator users who can view but not access camera passwords or change any state * workers that update signal state based on cameras' built-in motion detection or a security system's events but don't need to view videos * control over what can be done without authenticating Currently session permissions are just copied from user permissions, but you can also imagine admin sessions vs not, as a checkbox when signing in. This would match the standard Unix workflow of using a non-administrative session most of the time. Relevant to my current signals work (#28) and to the addition of an administrative API (#35, including #66).
2019-06-19 15:17:50 -07:00
}
give downloaded .mp4s a useful filename This is effective both for Chrome's "Save As" dialog and for curl -OJ. It makes the filename like 20190717135519-driveway-main.mp4 rather than view.mp4 (Chrome) or view.mp4?s=33-36&ts=true (Curl).
2020-02-16 23:16:19 -08:00
let (stream_id, camera_name);
{
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 14:34:00 -08:00
let db = self.db.lock();
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
let camera = db.get_camera(uuid).ok_or_else(|| {
plain_response(StatusCode::NOT_FOUND, format!("no such camera {}", uuid))
})?;
give downloaded .mp4s a useful filename This is effective both for Chrome's "Save As" dialog and for curl -OJ. It makes the filename like 20190717135519-driveway-main.mp4 rather than view.mp4 (Chrome) or view.mp4?s=33-36&ts=true (Curl).
2020-02-16 23:16:19 -08:00
camera_name = camera.short_name.clone();
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
stream_id = camera.streams[stream_type.index()].ok_or_else(|| {
plain_response(
StatusCode::NOT_FOUND,
format!("no such stream {}/{}", uuid, stream_type),
)
})?;
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 14:34:00 -08:00
};
give downloaded .mp4s a useful filename This is effective both for Chrome's "Save As" dialog and for curl -OJ. It makes the filename like 20190717135519-driveway-main.mp4 rather than view.mp4 (Chrome) or view.mp4?s=33-36&ts=true (Curl).
2020-02-16 23:16:19 -08:00
let mut start_time_for_filename = None;
backend support for live stream (#59) This is so far completely untested, for use by a new UI prototype. It creates a new URL endpoint which sends one video/mp4 media segment per key frame, with the dependent frames included. This means there will be about one key frame interval of latency (typically about a second). This seems hard to avoid, as mentioned in issue #59.
2019-01-21 15:58:52 -08:00
let mut builder = mp4::FileBuilder::new(mp4_type);
schema version 2: support sub streams This allows each camera to have a main and a sub stream. Previously there was a field in the schema for the sub stream's url, but it didn't do anything. Now you can configure individual retention for main and sub streams. They show up grouped in the UI. No support for upgrading from schema version 1 yet.
2018-01-23 11:05:07 -08:00
if let Some(q) = req.uri().query() {
upgrade to async hyper serve_generated_bytes is >3X faster. One caveat is that the reactor thread may stall when reading from the memory-mapped slice. Moonfire NVR is basically a single-user program, so that may not be so bad, but we'll see.
2017-03-02 19:29:28 -08:00
for (key, value) in form_urlencoded::parse(q.as_bytes()) {
let (key, value) = (key.borrow(), value.borrow());
match key {
"s" => {
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
let s = Segments::from_str(value).map_err(|()| {
plain_response(
StatusCode::BAD_REQUEST,
format!("invalid s parameter: {}", value),
)
})?;
logging improvements * allow debug/trace logging on release builds again * enable log messages from hyper. I didn't notice they went away with 0.14.0, although there's a breaking change in the log: https://github.com/hyperium/hyper/blob/master/CHANGELOG.md#v0140-2020-12-23 * downgrade some particularly spammy messages
2021-06-04 23:28:48 -07:00
trace!("stream_view_mp4: appending s={:?}", s);
start splitting wall and media duration for #34 This splits the schema and playback path. The recording path still adjusts the frame durations and always says the wall and media durations are the same. I expect to change that in a following commit. I wouldn't be surprised if that shakes out some bugs in this portion.
2020-08-04 21:44:01 -07:00
let mut est_segments = usize::try_from(s.ids.end - s.ids.start).unwrap();
upgrade to async hyper serve_generated_bytes is >3X faster. One caveat is that the reactor thread may stall when reading from the memory-mapped slice. Moonfire NVR is basically a single-user program, so that may not be so bad, but we'll see.
2017-03-02 19:29:28 -08:00
if let Some(end) = s.end_time {
// There should be roughly ceil((end - start) /
// desired_recording_duration) recordings in the desired timespan if
// there are no gaps or overlap, possibly another for misalignment of
// the requested timespan with the rotate offset and another because
// rotation only happens at key frames.
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
let ceil_durations = (end - s.start_time
+ recording::DESIRED_RECORDING_WALL_DURATION
- 1)
/ recording::DESIRED_RECORDING_WALL_DURATION;
upgrade to async hyper serve_generated_bytes is >3X faster. One caveat is that the reactor thread may stall when reading from the memory-mapped slice. Moonfire NVR is basically a single-user program, so that may not be so bad, but we'll see.
2017-03-02 19:29:28 -08:00
est_segments = cmp::min(est_segments, (ceil_durations + 2) as usize);
}
builder.reserve(est_segments);
let db = self.db.lock();
start splitting wall and media duration for #34 This splits the schema and playback path. The recording path still adjusts the frame durations and always says the wall and media durations are the same. I expect to change that in a following commit. I wouldn't be surprised if that shakes out some bugs in this portion.
2020-08-04 21:44:01 -07:00
let mut prev = None; // previous recording id
upgrade to async hyper serve_generated_bytes is >3X faster. One caveat is that the reactor thread may stall when reading from the memory-mapped slice. Moonfire NVR is basically a single-user program, so that may not be so bad, but we'll see.
2017-03-02 19:29:28 -08:00
let mut cur_off = 0;
take FnMut closures by reference I mistakenly thought these had to be monomorphized. (The FnOnce still does, until rust-lang/rfcs#1909 is implemented.) Turns out this way works fine. It should result in less compile time / code size, though I didn't check this.
2018-02-23 09:19:42 -08:00
db.list_recordings_by_id(stream_id, s.ids.clone(), &mut |r| {
reorganize the sample file directory The filenames now represent composite ids (stream id + recording id) rather than a separate uuid system with its own reservation for a few benefits: * This provides more information when there are inconsistencies. * This avoids the need for managing the reservations during recording. I expect this to simplify delaying flushing of newly written sample files. Now the directory has to be scanned at startup for files that never got written to the database, but that's acceptably fast even with millions of files. * Less information to keep in memory and in the recording_playback table. I'd considered using one directory per stream, which might help if the filesystem has trouble coping with huge directories. But that would mean each dir has to be fsync()ed separately (more latency and/or more multithreading). So I'll stick with this until I see concrete evidence of a problem that would solve. Test coverage of the error conditions is poor. I plan to do some restructuring of the db/dir code, hopefully making steps toward testability along the way.
2018-02-20 10:11:10 -08:00
let recording_id = r.id.recording();
allow listing and viewing uncommitted recordings There may be considerable lag between being fully written and being committed when using the flush_if_sec feature. Additionally, this is a step toward listing and viewing recordings before they're fully written. That's a considerable delay: 60 to 120 seconds for the first recording of a run, 0 to 60 seconds for subsequent recordings. These recordings aren't yet included in the information returned by /api/?days=true. They probably should be, but small steps.
2018-03-02 11:38:11 -08:00
if let Some(o) = s.open_id {
if r.open_id != o {
use typed errors in /view.mp4 path This fixes #178. Before, everything got translated to 5xx status; now it produces the correct type in several cases. Ideally I'd get rid of the untyped errors in all of web.rs; this is a small step.
2021-10-21 10:25:37 -07:00
bail_t!(
NotFound,
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
"recording {} has open id {}, requested {}",
r.id,
r.open_id,
o
);
allow listing and viewing uncommitted recordings There may be considerable lag between being fully written and being committed when using the flush_if_sec feature. Additionally, this is a step toward listing and viewing recordings before they're fully written. That's a considerable delay: 60 to 120 seconds for the first recording of a run, 0 to 60 seconds for subsequent recordings. These recordings aren't yet included in the information returned by /api/?days=true. They probably should be, but small steps.
2018-03-02 11:38:11 -08:00
}
}
upgrade to async hyper serve_generated_bytes is >3X faster. One caveat is that the reactor thread may stall when reading from the memory-mapped slice. Moonfire NVR is basically a single-user program, so that may not be so bad, but we'll see.
2017-03-02 19:29:28 -08:00
// Check for missing recordings.
match prev {
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
None if recording_id == s.ids.start => {}
use typed errors in /view.mp4 path This fixes #178. Before, everything got translated to 5xx status; now it produces the correct type in several cases. Ideally I'd get rid of the untyped errors in all of web.rs; this is a small step.
2021-10-21 10:25:37 -07:00
None => bail_t!(
NotFound,
"no such recording {}/{}",
stream_id,
s.ids.start
),
reorganize the sample file directory The filenames now represent composite ids (stream id + recording id) rather than a separate uuid system with its own reservation for a few benefits: * This provides more information when there are inconsistencies. * This avoids the need for managing the reservations during recording. I expect this to simplify delaying flushing of newly written sample files. Now the directory has to be scanned at startup for files that never got written to the database, but that's acceptably fast even with millions of files. * Less information to keep in memory and in the recording_playback table. I'd considered using one directory per stream, which might help if the filesystem has trouble coping with huge directories. But that would mean each dir has to be fsync()ed separately (more latency and/or more multithreading). So I'll stick with this until I see concrete evidence of a problem that would solve. Test coverage of the error conditions is poor. I plan to do some restructuring of the db/dir code, hopefully making steps toward testability along the way.
2018-02-20 10:11:10 -08:00
Some(id) if r.id.recording() != id + 1 => {
use typed errors in /view.mp4 path This fixes #178. Before, everything got translated to 5xx status; now it produces the correct type in several cases. Ideally I'd get rid of the untyped errors in all of web.rs; this is a small step.
2021-10-21 10:25:37 -07:00
bail_t!(NotFound, "no such recording {}/{}", stream_id, id + 1);
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
}
_ => {}
upgrade to async hyper serve_generated_bytes is >3X faster. One caveat is that the reactor thread may stall when reading from the memory-mapped slice. Moonfire NVR is basically a single-user program, so that may not be so bad, but we'll see.
2017-03-02 19:29:28 -08:00
};
reorganize the sample file directory The filenames now represent composite ids (stream id + recording id) rather than a separate uuid system with its own reservation for a few benefits: * This provides more information when there are inconsistencies. * This avoids the need for managing the reservations during recording. I expect this to simplify delaying flushing of newly written sample files. Now the directory has to be scanned at startup for files that never got written to the database, but that's acceptably fast even with millions of files. * Less information to keep in memory and in the recording_playback table. I'd considered using one directory per stream, which might help if the filesystem has trouble coping with huge directories. But that would mean each dir has to be fsync()ed separately (more latency and/or more multithreading). So I'll stick with this until I see concrete evidence of a problem that would solve. Test coverage of the error conditions is poor. I plan to do some restructuring of the db/dir code, hopefully making steps toward testability along the way.
2018-02-20 10:11:10 -08:00
prev = Some(recording_id);
upgrade to async hyper serve_generated_bytes is >3X faster. One caveat is that the reactor thread may stall when reading from the memory-mapped slice. Moonfire NVR is basically a single-user program, so that may not be so bad, but we'll see.
2017-03-02 19:29:28 -08:00
// Add a segment for the relevant part of the recording, if any.
start splitting wall and media duration for #34 This splits the schema and playback path. The recording path still adjusts the frame durations and always says the wall and media durations are the same. I expect to change that in a following commit. I wouldn't be surprised if that shakes out some bugs in this portion.
2020-08-04 21:44:01 -07:00
// Note all calculations here are in wall times / wall durations.
upgrade to async hyper serve_generated_bytes is >3X faster. One caveat is that the reactor thread may stall when reading from the memory-mapped slice. Moonfire NVR is basically a single-user program, so that may not be so bad, but we'll see.
2017-03-02 19:29:28 -08:00
let end_time = s.end_time.unwrap_or(i64::max_value());
fix live view This broke with the media vs wall duration split, part of #34.
2020-08-07 10:16:06 -07:00
let wd = i64::from(r.wall_duration_90k);
if s.start_time <= cur_off + wd && cur_off < end_time {
upgrade to async hyper serve_generated_bytes is >3X faster. One caveat is that the reactor thread may stall when reading from the memory-mapped slice. Moonfire NVR is basically a single-user program, so that may not be so bad, but we'll see.
2017-03-02 19:29:28 -08:00
let start = cmp::max(0, s.start_time - cur_off);
fix live view This broke with the media vs wall duration split, part of #34.
2020-08-07 10:16:06 -07:00
let end = cmp::min(wd, end_time - cur_off);
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
let wr = i32::try_from(start).unwrap()..i32::try_from(end).unwrap();
logging improvements * allow debug/trace logging on release builds again * enable log messages from hyper. I didn't notice they went away with 0.14.0, although there's a breaking change in the log: https://github.com/hyperium/hyper/blob/master/CHANGELOG.md#v0140-2020-12-23 * downgrade some particularly spammy messages
2021-06-04 23:28:48 -07:00
trace!(
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
"...appending recording {} with wall duration {:?} \
(out of total {})",
logging improvements * allow debug/trace logging on release builds again * enable log messages from hyper. I didn't notice they went away with 0.14.0, although there's a breaking change in the log: https://github.com/hyperium/hyper/blob/master/CHANGELOG.md#v0140-2020-12-23 * downgrade some particularly spammy messages
2021-06-04 23:28:48 -07:00
r.id,
wr,
wd
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
);
give downloaded .mp4s a useful filename This is effective both for Chrome's "Save As" dialog and for curl -OJ. It makes the filename like 20190717135519-driveway-main.mp4 rather than view.mp4 (Chrome) or view.mp4?s=33-36&ts=true (Curl).
2020-02-16 23:16:19 -08:00
if start_time_for_filename.is_none() {
start_time_for_filename =
Some(r.start + recording::Duration(start));
}
fix live view This broke with the media vs wall duration split, part of #34.
2020-08-07 10:16:06 -07:00
use recording::rescale;
let mr =
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
rescale(wr.start, r.wall_duration_90k, r.media_duration_90k)
..rescale(
wr.end,
r.wall_duration_90k,
r.media_duration_90k,
);
live stream frame-by-frame rather than GOP-by-GOP (#59) This should reduce live stream latency by two seconds when my cameras are at their default setting (I frame interval = 2 * frame rate)! I was under the impression that every HTML5 Media Source Extensions media segment had to start with a Random Access Point. This used to be true, but apparently changed quite a while ago: https://bugs.chromium.org/p/chromium/issues/detail?id=229412 Support generating segments that don't start with a key frame, and plumb this through the mp4 media segment generation logic. Add some extra error checking in mp4 slice handling, as my first attempts had a mismatch between expected and actual lengths that silently returned corrupted .m4s files. Also pull everything from the most recent key frame on along with the first live segment to reduce startup latency. Live view is quite a bit more pleasant now.
2020-08-07 15:30:22 -07:00
builder.append(&db, r, mr, true)?;
upgrade to async hyper serve_generated_bytes is >3X faster. One caveat is that the reactor thread may stall when reading from the memory-mapped slice. Moonfire NVR is basically a single-user program, so that may not be so bad, but we'll see.
2017-03-02 19:29:28 -08:00
} else {
logging improvements * allow debug/trace logging on release builds again * enable log messages from hyper. I didn't notice they went away with 0.14.0, although there's a breaking change in the log: https://github.com/hyperium/hyper/blob/master/CHANGELOG.md#v0140-2020-12-23 * downgrade some particularly spammy messages
2021-06-04 23:28:48 -07:00
trace!("...skipping recording {} wall dur {}", r.id, wd);
upgrade to async hyper serve_generated_bytes is >3X faster. One caveat is that the reactor thread may stall when reading from the memory-mapped slice. Moonfire NVR is basically a single-user program, so that may not be so bad, but we'll see.
2017-03-02 19:29:28 -08:00
}
fix live view This broke with the media vs wall duration split, part of #34.
2020-08-07 10:16:06 -07:00
cur_off += wd;
upgrade to async hyper serve_generated_bytes is >3X faster. One caveat is that the reactor thread may stall when reading from the memory-mapped slice. Moonfire NVR is basically a single-user program, so that may not be so bad, but we'll see.
2017-03-02 19:29:28 -08:00
Ok(())
use typed errors in /view.mp4 path This fixes #178. Before, everything got translated to 5xx status; now it produces the correct type in several cases. Ideally I'd get rid of the untyped errors in all of web.rs; this is a small step.
2021-10-21 10:25:37 -07:00
})?;
upgrade to async hyper serve_generated_bytes is >3X faster. One caveat is that the reactor thread may stall when reading from the memory-mapped slice. Moonfire NVR is basically a single-user program, so that may not be so bad, but we'll see.
2017-03-02 19:29:28 -08:00
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-20 22:08:18 -08:00
// Check for missing recordings.
match prev {
upgrade to async hyper serve_generated_bytes is >3X faster. One caveat is that the reactor thread may stall when reading from the memory-mapped slice. Moonfire NVR is basically a single-user program, so that may not be so bad, but we'll see.
2017-03-02 19:29:28 -08:00
Some(id) if s.ids.end != id + 1 => {
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
return Err(not_found(format!(
"no such recording {}/{}",
stream_id,
s.ids.end - 1
)));
}
upgrade to async hyper serve_generated_bytes is >3X faster. One caveat is that the reactor thread may stall when reading from the memory-mapped slice. Moonfire NVR is basically a single-user program, so that may not be so bad, but we'll see.
2017-03-02 19:29:28 -08:00
None => {
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
return Err(not_found(format!(
"no such recording {}/{}",
stream_id, s.ids.start
)));
}
_ => {}
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-20 22:08:18 -08:00
};
upgrade to async hyper serve_generated_bytes is >3X faster. One caveat is that the reactor thread may stall when reading from the memory-mapped slice. Moonfire NVR is basically a single-user program, so that may not be so bad, but we'll see.
2017-03-02 19:29:28 -08:00
if let Some(end) = s.end_time {
if end > cur_off {
small readability improvements to web.rs I think this has a minor behavior change: permission denied replies change to HTTP 403 where they were HTTP 401. The new behavior seems more correct, as these errors can occur when authentication has succeeded but the session in question is not authorized for the given operation. The UI currently doesn't care about this distinction.
2021-03-10 16:18:47 -08:00
bail_t!(
InvalidArgument,
"end time {} is beyond specified recordings",
end
);
upgrade to async hyper serve_generated_bytes is >3X faster. One caveat is that the reactor thread may stall when reading from the memory-mapped slice. Moonfire NVR is basically a single-user program, so that may not be so bad, but we'll see.
2017-03-02 19:29:28 -08:00
}
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-20 22:08:18 -08:00
}
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
}
"ts" => builder
.include_timestamp_subtitle_track(value == "true")
.map_err(from_base_error)?,
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
_ => return Err(bad_req(format!("parameter {} not understood", key))),
upgrade to async hyper serve_generated_bytes is >3X faster. One caveat is that the reactor thread may stall when reading from the memory-mapped slice. Moonfire NVR is basically a single-user program, so that may not be so bad, but we'll see.
2017-03-02 19:29:28 -08:00
}
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
}
upgrade to async hyper serve_generated_bytes is >3X faster. One caveat is that the reactor thread may stall when reading from the memory-mapped slice. Moonfire NVR is basically a single-user program, so that may not be so bad, but we'll see.
2017-03-02 19:29:28 -08:00
}
give downloaded .mp4s a useful filename This is effective both for Chrome's "Save As" dialog and for curl -OJ. It makes the filename like 20190717135519-driveway-main.mp4 rather than view.mp4 (Chrome) or view.mp4?s=33-36&ts=true (Curl).
2020-02-16 23:16:19 -08:00
if let Some(start) = start_time_for_filename {
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
let tm = time::at(time::Timespec {
sec: start.unix_seconds(),
nsec: 0,
});
address some no-op clippy warnings
2021-05-17 14:31:50 -07:00
let stream_abbrev = if stream_type == db::StreamType::Main {
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
"main"
} else {
"sub"
};
let suffix = if mp4_type == mp4::Type::Normal {
"mp4"
} else {
"m4s"
};
builder
.set_filename(&format!(
"{}-{}-{}.{}",
tm.strftime("%Y%m%d%H%M%S").unwrap(),
camera_name,
stream_abbrev,
suffix
))
give downloaded .mp4s a useful filename This is effective both for Chrome's "Save As" dialog and for curl -OJ. It makes the filename like 20190717135519-driveway-main.mp4 rather than view.mp4 (Chrome) or view.mp4?s=33-36&ts=true (Curl).
2020-02-16 23:16:19 -08:00
.map_err(from_base_error)?;
}
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
let mp4 = builder
.build(self.db.clone(), self.dirs_by_stream_id.clone())
.map_err(from_base_error)?;
add a url for getting debug info about a .mp4 file and add a unit test of path decoding along the way
2018-12-29 13:06:44 -06:00
if debug {
return Ok(plain_response(StatusCode::OK, format!("{:#?}", mp4)));
}
crate rename: http-(entity|file) -> http-serve
2018-01-23 11:08:21 -08:00
Ok(http_serve::serve(mp4, req))
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 14:34:00 -08:00
}
add a basic Javascript UI The Javascript is pretty amateurish I'm sure but at least it's something to iterate from. It's already much more pleasant for browsing through videos in several ways: * more responsive to load only a day at a time rather than 90+ days * much easier to see the same time segment on several cameras * more pleasant to have the videos load as a popup rather than a link that blows away your position in an enormous list * exposes the fancier .mp4 generation options: splitting at lengths other than the default, trimming to an arbitrary start and end time, including a subtitle track with timestamps. There's a slight regression in functionality: I didn't match the former top-level page which showed how much camera used of its disk allocation and the total duration of video. This is exposed in the JSON API, so it shouldn't be too hard to add back.
2017-10-21 21:54:27 -07:00
UI preferences: #153 #155
2021-09-01 15:01:42 -07:00
async fn user(&self, req: Request<hyper::Body>, caller: Caller, id: i32) -> ResponseResult {
if caller.user.map(|u| u.id) != Some(id) {
bail_t!(Unauthenticated, "must be authenticated as supplied user");
}
match *req.method() {
Method::POST => self.post_user(req, id).await,
_ => Err(plain_response(StatusCode::METHOD_NOT_ALLOWED, "POST expected").into()),
}
}
async fn post_user(&self, mut req: Request<hyper::Body>, id: i32) -> ResponseResult {
let r = extract_json_body(&mut req).await?;
let r: json::PostUser = serde_json::from_slice(&r).map_err(|e| bad_req(e.to_string()))?;
let mut db = self.db.lock();
let user = db
.users_by_id()
.get(&id)
.ok_or_else(|| format_err_t!(Internal, "can't find currently authenticated user"))?;
if let Some(precondition) = r.precondition {
add config json to user table
2021-10-26 13:08:45 -07:00
if matches!(precondition.preferences, Some(p) if p != user.config.preferences) {
UI preferences: #153 #155
2021-09-01 15:01:42 -07:00
bail_t!(FailedPrecondition, "preferences mismatch");
}
}
if let Some(update) = r.update {
let mut change = user.change();
if let Some(preferences) = update.preferences {
add config json to user table
2021-10-26 13:08:45 -07:00
change.config.preferences = preferences;
UI preferences: #153 #155
2021-09-01 15:01:42 -07:00
}
db.apply_user_change(change).map_err(internal_server_err)?;
}
Ok(plain_response(StatusCode::NO_CONTENT, &b""[..]))
}
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
fn authreq(&self, req: &Request<::hyper::Body>) -> auth::Request {
auth::Request {
when_sec: Some(self.db.clocks().realtime().sec),
support proxy forwarded headers I went with legacy headers (X-Real-IP, X-Forwarded-Proto) because they appear to be more widely supported than the RFC 7239 Forwarded header.
2018-11-28 14:22:30 -08:00
addr: if self.trust_forward_hdrs {
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
req.headers()
.get("X-Real-IP")
.and_then(|v| v.to_str().ok())
.and_then(|v| IpAddr::from_str(v).ok())
} else {
None
},
user_agent: req
.headers()
.get(header::USER_AGENT)
.map(|ua| ua.as_bytes().to_vec()),
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
}
}
endpoint to debug --trust-forward-hdrs for #26
2018-12-01 00:44:19 -08:00
fn request(&self, req: &Request<::hyper::Body>) -> ResponseResult {
let authreq = self.authreq(req);
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
let host = req
.headers()
.get(header::HOST)
.map(|h| String::from_utf8_lossy(h.as_bytes()));
let agent = authreq
.user_agent
.as_ref()
.map(|u| String::from_utf8_lossy(&u[..]));
Ok(plain_response(
StatusCode::OK,
format!(
"when: {}\n\
document proxy setup in guide/secure.md (for #26) The guide is not as quick to follow and amateur-friendly as I'd like. A few things that might improve matters: * complete #27 (built-in https+letsencrypt), so that when not sharing the port, users don't need to use nginx or certbot. * more ubiquitous IPv6 (out of my control but should happen over time) to reduce need to share the port * embed a dynamic DNS client * support UPnP Internet Gateway Device Control Protocol (if common routers have this enabled? probably not for security reasons.) It's progress, though. Enough that I think I'll merge the auth branch into master shortly.
2018-12-27 16:00:15 -06:00
host: {:?}\n\
endpoint to debug --trust-forward-hdrs for #26
2018-12-01 00:44:19 -08:00
addr: {:?}\n\
user_agent: {:?}\n\
secure: {:?}",
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
time::at(time::Timespec {
sec: authreq.when_sec.unwrap(),
nsec: 0
})
.strftime("%FT%T")
.map(|f| f.to_string())
.unwrap_or_else(|e| e.to_string()),
cleanup some awkward Option call chains
2021-05-06 09:47:07 -07:00
host.as_deref(),
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
&authreq.addr,
cleanup some awkward Option call chains
2021-05-06 09:47:07 -07:00
agent.as_deref(),
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
self.is_secure(req)
),
))
endpoint to debug --trust-forward-hdrs for #26
2018-12-01 00:44:19 -08:00
}
small readability improvements to web.rs I think this has a minor behavior change: permission denied replies change to HTTP 403 where they were HTTP 401. The new behavior seems more correct, as these errors can occur when authentication has succeeded but the session in question is not authorized for the given operation. The UI currently doesn't care about this distinction.
2021-03-10 16:18:47 -08:00
/// Returns true iff the client is connected over `https`.
/// Moonfire NVR currently doesn't directly serve `https`, but it supports
/// proxies which set the `X-Forwarded-Proto` header. See `guide/secure.md`
/// for more information.
support proxy forwarded headers I went with legacy headers (X-Real-IP, X-Forwarded-Proto) because they appear to be more widely supported than the RFC 7239 Forwarded header.
2018-11-28 14:22:30 -08:00
fn is_secure(&self, req: &Request<::hyper::Body>) -> bool {
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
self.trust_forward_hdrs
&& req
.headers()
.get("X-Forwarded-Proto")
.map(|v| v.as_bytes() == b"https")
.unwrap_or(false)
support proxy forwarded headers I went with legacy headers (X-Real-IP, X-Forwarded-Proto) because they appear to be more widely supported than the RFC 7239 Forwarded header.
2018-11-28 14:22:30 -08:00
}
more readable async web code This uses "async fn" throughout rather than a mix of async and the older futures style. And it takes advantage of the "self: Arc<Self>" syntax to avoid having a ServiceInner. It was confusing to have some methods on Service and some on ServiceInner; now that distinction is gone. One downside is there's a little more atomic reference-counting. Before, service_fn essentially took an &Arc<Self>, which means it could call Arc::clone where its use of self actually outlived the future (see stream_live_m4s) but didn't need to otherwise. After, it calls an async fn that takes Arc<Self>. Using &Arc<Self> is apparently possible (as of Rust 1.41) but using that with "async fn" means the returned future is tied to its lifetime. The workaround is to use async blocks as described here: <https://rust-lang.github.io/async-book/03_async_await/01_chapter.html> but that's really ugly: it brings back the explicit Future reference, requires futures::future::Either in some cases, and introduces another level of indenting. I think it's better to just pay the arc costs which are probably negligible, or at least cheaper than the boxing was before. Oh, and I make this compile on Rust 1.40 again as it claimed to. http-serve accidentally used the &Arc<Self> thing which broke this. Update to a freshly-pushed commit which doesn't do that.
2020-05-30 19:08:54 -07:00
async fn login(&self, mut req: Request<::hyper::Body>) -> ResponseResult {
let r = extract_json_body(&mut req).await?;
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
let r: json::LoginRequest =
serde_json::from_slice(&r).map_err(|e| bad_req(e.to_string()))?;
more readable async web code This uses "async fn" throughout rather than a mix of async and the older futures style. And it takes advantage of the "self: Arc<Self>" syntax to avoid having a ServiceInner. It was confusing to have some methods on Service and some on ServiceInner; now that distinction is gone. One downside is there's a little more atomic reference-counting. Before, service_fn essentially took an &Arc<Self>, which means it could call Arc::clone where its use of self actually outlived the future (see stream_live_m4s) but didn't need to otherwise. After, it calls an async fn that takes Arc<Self>. Using &Arc<Self> is apparently possible (as of Rust 1.41) but using that with "async fn" means the returned future is tied to its lifetime. The workaround is to use async blocks as described here: <https://rust-lang.github.io/async-book/03_async_await/01_chapter.html> but that's really ugly: it brings back the explicit Future reference, requires futures::future::Either in some cases, and introduces another level of indenting. I think it's better to just pay the arc costs which are probably negligible, or at least cheaper than the boxing was before. Oh, and I make this compile on Rust 1.40 again as it claimed to. http-serve accidentally used the &Arc<Self> thing which broke this. Update to a freshly-pushed commit which doesn't do that.
2020-05-30 19:08:54 -07:00
let authreq = self.authreq(&req);
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
let host = req
.headers()
.get(header::HOST)
.ok_or_else(|| bad_req("missing Host header!"))?;
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
let host = host.as_bytes();
overhaul HTTP serving and caching * use content-hashed paths for static resources (except the top-level request), with immutable Cache-Control headers. This should improve cache behavior in both directions: avoid preventable HTTP requests and cause immediate refresh when needed. I had some staleness when browsing with my phone. * set up the favicons properly while I'm at it (closes #50). I used the convenient favicons-webpack-plugin to build everything from a .svg. I've hit an error similar to lovell/sharp#1593 at least once though so I might change my mind about that part if it continues to be problematic. * use http-serve's new directory traversal code for static file serving. This removes the odd behavior where files that weren't present at server startup couldn't be served. (I wasn't comfortable switching to the content-hashed paths before doing this.) It also means the static files can be served compressed. JSON API responses were already served compressed, so this closes #25. * for a given API URL, decide if we want it to be cached or not server-side. Stop using jQuery's kludgy cache-defeating _=<timestamp> URL parameter. I might start setting etags on some of these things and could serve 304 Not Modified responses if it's genuinely unmodified.
2020-05-29 21:20:14 -07:00
let domain = match memchr(b':', host) {
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
Some(colon) => &host[0..colon],
None => host,
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
}
.to_owned();
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
let mut l = self.db.lock();
use SameSite=Lax instead of SameSite=Strict To improve reliability of live streams (#59) on Safari. Safari was dropping the cookie from websocket update requests. (But it worked sometimes. I don't get why.) I saw folks on the Internet thinking this related to HttpOnly: * https://developer.apple.com/forums/thread/104488 * https://stackoverflow.com/q/47742807/23584 but I still see this behavior without HttpOnly. SameSite=Strict vs SameSite=Lax appears to make a difference. Try that instead. SameSite=Strict is pointless for us anyway as noted in a new comment. Turning off HttpOnly would be more unfortunate security-wise.
2021-03-31 10:44:08 -07:00
// If the request came in over https, tell the browser to only send the cookie on https
// requests also.
more readable async web code This uses "async fn" throughout rather than a mix of async and the older futures style. And it takes advantage of the "self: Arc<Self>" syntax to avoid having a ServiceInner. It was confusing to have some methods on Service and some on ServiceInner; now that distinction is gone. One downside is there's a little more atomic reference-counting. Before, service_fn essentially took an &Arc<Self>, which means it could call Arc::clone where its use of self actually outlived the future (see stream_live_m4s) but didn't need to otherwise. After, it calls an async fn that takes Arc<Self>. Using &Arc<Self> is apparently possible (as of Rust 1.41) but using that with "async fn" means the returned future is tied to its lifetime. The workaround is to use async blocks as described here: <https://rust-lang.github.io/async-book/03_async_await/01_chapter.html> but that's really ugly: it brings back the explicit Future reference, requires futures::future::Either in some cases, and introduces another level of indenting. I think it's better to just pay the arc costs which are probably negligible, or at least cheaper than the boxing was before. Oh, and I make this compile on Rust 1.40 again as it claimed to. http-serve accidentally used the &Arc<Self> thing which broke this. Update to a freshly-pushed commit which doesn't do that.
2020-05-30 19:08:54 -07:00
let is_secure = self.is_secure(&req);
use SameSite=Lax instead of SameSite=Strict To improve reliability of live streams (#59) on Safari. Safari was dropping the cookie from websocket update requests. (But it worked sometimes. I don't get why.) I saw folks on the Internet thinking this related to HttpOnly: * https://developer.apple.com/forums/thread/104488 * https://stackoverflow.com/q/47742807/23584 but I still see this behavior without HttpOnly. SameSite=Strict vs SameSite=Lax appears to make a difference. Try that instead. SameSite=Strict is pointless for us anyway as noted in a new comment. Turning off HttpOnly would be more unfortunate security-wise.
2021-03-31 10:44:08 -07:00
// Use SameSite=Lax rather than SameSite=Strict. Safari apparently doesn't send
// SameSite=Strict cookies on WebSocket upgrade requests. There's no real security
// difference for Moonfire NVR anyway. SameSite=Strict exists as CSRF protection for
// sites that (unlike Moonfire NVR) don't follow best practices by (a)
// mutating based on GET requests and (b) not using CSRF tokens.
use auth::SessionFlag;
let flags = (SessionFlag::HttpOnly as i32)
| (SessionFlag::SameSite as i32)
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
| if is_secure {
use SameSite=Lax instead of SameSite=Strict To improve reliability of live streams (#59) on Safari. Safari was dropping the cookie from websocket update requests. (But it worked sometimes. I don't get why.) I saw folks on the Internet thinking this related to HttpOnly: * https://developer.apple.com/forums/thread/104488 * https://stackoverflow.com/q/47742807/23584 but I still see this behavior without HttpOnly. SameSite=Strict vs SameSite=Lax appears to make a difference. Try that instead. SameSite=Strict is pointless for us anyway as noted in a new comment. Turning off HttpOnly would be more unfortunate security-wise.
2021-03-31 10:44:08 -07:00
SessionFlag::Secure as i32
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
} else {
0
};
let (sid, _) = l
.login_by_password(authreq, &r.username, r.password, Some(domain), flags)
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
.map_err(|e| plain_response(StatusCode::UNAUTHORIZED, e.to_string()))?;
use SameSite=Lax instead of SameSite=Strict To improve reliability of live streams (#59) on Safari. Safari was dropping the cookie from websocket update requests. (But it worked sometimes. I don't get why.) I saw folks on the Internet thinking this related to HttpOnly: * https://developer.apple.com/forums/thread/104488 * https://stackoverflow.com/q/47742807/23584 but I still see this behavior without HttpOnly. SameSite=Strict vs SameSite=Lax appears to make a difference. Try that instead. SameSite=Strict is pointless for us anyway as noted in a new comment. Turning off HttpOnly would be more unfortunate security-wise.
2021-03-31 10:44:08 -07:00
let cookie = encode_sid(sid, flags);
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
Ok(Response::builder()
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
.header(
header::SET_COOKIE,
use SameSite=Lax instead of SameSite=Strict To improve reliability of live streams (#59) on Safari. Safari was dropping the cookie from websocket update requests. (But it worked sometimes. I don't get why.) I saw folks on the Internet thinking this related to HttpOnly: * https://developer.apple.com/forums/thread/104488 * https://stackoverflow.com/q/47742807/23584 but I still see this behavior without HttpOnly. SameSite=Strict vs SameSite=Lax appears to make a difference. Try that instead. SameSite=Strict is pointless for us anyway as noted in a new comment. Turning off HttpOnly would be more unfortunate security-wise.
2021-03-31 10:44:08 -07:00
HeaderValue::try_from(cookie).expect("cookie can't have invalid bytes"),
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
)
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
.status(StatusCode::NO_CONTENT)
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
.body(b""[..].into())
.unwrap())
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
}
more readable async web code This uses "async fn" throughout rather than a mix of async and the older futures style. And it takes advantage of the "self: Arc<Self>" syntax to avoid having a ServiceInner. It was confusing to have some methods on Service and some on ServiceInner; now that distinction is gone. One downside is there's a little more atomic reference-counting. Before, service_fn essentially took an &Arc<Self>, which means it could call Arc::clone where its use of self actually outlived the future (see stream_live_m4s) but didn't need to otherwise. After, it calls an async fn that takes Arc<Self>. Using &Arc<Self> is apparently possible (as of Rust 1.41) but using that with "async fn" means the returned future is tied to its lifetime. The workaround is to use async blocks as described here: <https://rust-lang.github.io/async-book/03_async_await/01_chapter.html> but that's really ugly: it brings back the explicit Future reference, requires futures::future::Either in some cases, and introduces another level of indenting. I think it's better to just pay the arc costs which are probably negligible, or at least cheaper than the boxing was before. Oh, and I make this compile on Rust 1.40 again as it claimed to. http-serve accidentally used the &Arc<Self> thing which broke this. Update to a freshly-pushed commit which doesn't do that.
2020-05-30 19:08:54 -07:00
async fn logout(&self, mut req: Request<hyper::Body>) -> ResponseResult {
let r = extract_json_body(&mut req).await?;
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
let r: json::LogoutRequest =
serde_json::from_slice(&r).map_err(|e| bad_req(e.to_string()))?;
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
let mut res = Response::new(b""[..].into());
more readable async web code This uses "async fn" throughout rather than a mix of async and the older futures style. And it takes advantage of the "self: Arc<Self>" syntax to avoid having a ServiceInner. It was confusing to have some methods on Service and some on ServiceInner; now that distinction is gone. One downside is there's a little more atomic reference-counting. Before, service_fn essentially took an &Arc<Self>, which means it could call Arc::clone where its use of self actually outlived the future (see stream_live_m4s) but didn't need to otherwise. After, it calls an async fn that takes Arc<Self>. Using &Arc<Self> is apparently possible (as of Rust 1.41) but using that with "async fn" means the returned future is tied to its lifetime. The workaround is to use async blocks as described here: <https://rust-lang.github.io/async-book/03_async_await/01_chapter.html> but that's really ugly: it brings back the explicit Future reference, requires futures::future::Either in some cases, and introduces another level of indenting. I think it's better to just pay the arc costs which are probably negligible, or at least cheaper than the boxing was before. Oh, and I make this compile on Rust 1.40 again as it claimed to. http-serve accidentally used the &Arc<Self> thing which broke this. Update to a freshly-pushed commit which doesn't do that.
2020-05-30 19:08:54 -07:00
if let Some(sid) = extract_sid(&req) {
let authreq = self.authreq(&req);
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
let mut l = self.db.lock();
let hash = sid.hash();
let need_revoke = match l.authenticate_session(authreq.clone(), &hash) {
Ok((s, _)) => {
use application/json for login and logout
2020-01-08 23:23:58 -08:00
if !csrf_matches(r.csrf, s.csrf()) {
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
warn!("logout request with missing/incorrect csrf");
return Err(bad_req("logout with incorrect csrf token"));
}
info!("revoking session");
true
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
}
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
Err(e) => {
// TODO: distinguish "no such session", "session is no longer valid", and
// "user ... is disabled" (which are all client error / bad state) from database
// errors.
warn!("logout failed: {}", e);
false
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
}
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
};
if need_revoke {
// TODO: inline this above with non-lexical lifetimes.
l.revoke_session(auth::RevocationReason::LoggedOut, None, authreq, &hash)
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
.map_err(internal_server_err)?;
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
}
// By now the session is invalid (whether it was valid to start with or not).
// Clear useless cookie.
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
res.headers_mut().append(
header::SET_COOKIE,
HeaderValue::from_str("s=; Max-Age=0; Path=/").unwrap(),
);
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
}
*res.status_mut() = StatusCode::NO_CONTENT;
Ok(res)
}
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
async fn post_signals(&self, mut req: Request<hyper::Body>, caller: Caller) -> ResponseResult {
add concept of user/session permissions (I also considered the names "capabilities" and "scopes", but I think "permissions" is the most widely understood.) This is increasingly necessary as the web API becomes more capable. Among other things, it allows: * non-administrator users who can view but not access camera passwords or change any state * workers that update signal state based on cameras' built-in motion detection or a security system's events but don't need to view videos * control over what can be done without authenticating Currently session permissions are just copied from user permissions, but you can also imagine admin sessions vs not, as a checkbox when signing in. This would match the standard Unix workflow of using a non-administrative session most of the time. Relevant to my current signals work (#28) and to the addition of an administrative API (#35, including #66).
2019-06-19 15:17:50 -07:00
if !caller.permissions.update_signals {
small readability improvements to web.rs I think this has a minor behavior change: permission denied replies change to HTTP 403 where they were HTTP 401. The new behavior seems more correct, as these errors can occur when authentication has succeeded but the session in question is not authorized for the given operation. The UI currently doesn't care about this distinction.
2021-03-10 16:18:47 -08:00
bail_t!(PermissionDenied, "update_signals required");
add concept of user/session permissions (I also considered the names "capabilities" and "scopes", but I think "permissions" is the most widely understood.) This is increasingly necessary as the web API becomes more capable. Among other things, it allows: * non-administrator users who can view but not access camera passwords or change any state * workers that update signal state based on cameras' built-in motion detection or a security system's events but don't need to view videos * control over what can be done without authenticating Currently session permissions are just copied from user permissions, but you can also imagine admin sessions vs not, as a checkbox when signing in. This would match the standard Unix workflow of using a non-administrative session most of the time. Relevant to my current signals work (#28) and to the addition of an administrative API (#35, including #66).
2019-06-19 15:17:50 -07:00
}
more readable async web code This uses "async fn" throughout rather than a mix of async and the older futures style. And it takes advantage of the "self: Arc<Self>" syntax to avoid having a ServiceInner. It was confusing to have some methods on Service and some on ServiceInner; now that distinction is gone. One downside is there's a little more atomic reference-counting. Before, service_fn essentially took an &Arc<Self>, which means it could call Arc::clone where its use of self actually outlived the future (see stream_live_m4s) but didn't need to otherwise. After, it calls an async fn that takes Arc<Self>. Using &Arc<Self> is apparently possible (as of Rust 1.41) but using that with "async fn" means the returned future is tied to its lifetime. The workaround is to use async blocks as described here: <https://rust-lang.github.io/async-book/03_async_await/01_chapter.html> but that's really ugly: it brings back the explicit Future reference, requires futures::future::Either in some cases, and introduces another level of indenting. I think it's better to just pay the arc costs which are probably negligible, or at least cheaper than the boxing was before. Oh, and I make this compile on Rust 1.40 again as it claimed to. http-serve accidentally used the &Arc<Self> thing which broke this. Update to a freshly-pushed commit which doesn't do that.
2020-05-30 19:08:54 -07:00
let r = extract_json_body(&mut req).await?;
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
let r: json::PostSignalsRequest =
serde_json::from_slice(&r).map_err(|e| bad_req(e.to_string()))?;
web api glue for updating signals This is very lightly tested, but it at least sometimes works.
2019-06-14 16:11:12 -07:00
let now = recording::Time::new(self.db.clocks().realtime());
refine timestamps in json signals api * API change: in update signals, allow setting a start time relative to now. This is an accuracy improvement in the case where the client has been retrying an initial request for a while. Kind of an obscure corner case but easy enough to address. And use a more convenient enum representation. * in update signals, choose `now` before acquiring the database lock. If lock acquisition takes a long time, this more accurately reflects the time the caller intended. * in general, make Time and Duration (de)serializable and use them in json types. This makes the types more self-describing, with better debug printing on both the server side and on the client library (in moonfire-playground). To make this work, base has to import serde which initially seemed like poor layering to me, but serde seems to be imported in some pretty foundational Rust crates for this reason. I'll go with it.
2021-04-21 10:44:01 -07:00
let mut l = self.db.lock();
let start = match r.start {
json::PostSignalsTimeBase::Epoch(t) => t,
json::PostSignalsTimeBase::Now(d) => now + d,
};
let end = match r.end {
json::PostSignalsTimeBase::Epoch(t) => t,
json::PostSignalsTimeBase::Now(d) => now + d,
web api glue for updating signals This is very lightly tested, but it at least sometimes works.
2019-06-14 16:11:12 -07:00
};
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
l.update_signals(start..end, &r.signal_ids, &r.states)
.map_err(from_base_error)?;
refine timestamps in json signals api * API change: in update signals, allow setting a start time relative to now. This is an accuracy improvement in the case where the client has been retrying an initial request for a while. Kind of an obscure corner case but easy enough to address. And use a more convenient enum representation. * in update signals, choose `now` before acquiring the database lock. If lock acquisition takes a long time, this more accurately reflects the time the caller intended. * in general, make Time and Duration (de)serializable and use them in json types. This makes the types more self-describing, with better debug printing on both the server side and on the client library (in moonfire-playground). To make this work, base has to import serde which initially seemed like poor layering to me, but serde seems to be imported in some pretty foundational Rust crates for this reason. I'll go with it.
2021-04-21 10:44:01 -07:00
serve_json(&req, &json::PostSignalsResponse { time_90k: now })
web api glue for updating signals This is very lightly tested, but it at least sometimes works.
2019-06-14 16:11:12 -07:00
}
fn get_signals(&self, req: &Request<hyper::Body>) -> ResponseResult {
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
let mut time = recording::Time::min_value()..recording::Time::max_value();
read-only signals support (#28) This is mostly untested and useless by itself, but it's a starting point. In particular: * there's no way to set up signals or add/remove/update events yet except by manual changes to the database. * if you associate a signal with a camera then remove the camera, hitting /api/ will error out.
2019-06-06 16:18:13 -07:00
if let Some(q) = req.uri().query() {
for (key, value) in form_urlencoded::parse(q.as_bytes()) {
let (key, value) = (key.borrow(), value.borrow());
match key {
"startTime90k" => {
time.start = recording::Time::parse(value)
.map_err(|_| bad_req("unparseable startTime90k"))?
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
}
read-only signals support (#28) This is mostly untested and useless by itself, but it's a starting point. In particular: * there's no way to set up signals or add/remove/update events yet except by manual changes to the database. * if you associate a signal with a camera then remove the camera, hitting /api/ will error out.
2019-06-06 16:18:13 -07:00
"endTime90k" => {
time.end = recording::Time::parse(value)
.map_err(|_| bad_req("unparseable endTime90k"))?
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
}
_ => {}
read-only signals support (#28) This is mostly untested and useless by itself, but it's a starting point. In particular: * there's no way to set up signals or add/remove/update events yet except by manual changes to the database. * if you associate a signal with a camera then remove the camera, hitting /api/ will error out.
2019-06-06 16:18:13 -07:00
}
}
}
let mut signals = json::Signals::default();
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
self.db
.lock()
.list_changes_by_time(time, &mut |c: &db::signal::ListStateChangesRow| {
refine timestamps in json signals api * API change: in update signals, allow setting a start time relative to now. This is an accuracy improvement in the case where the client has been retrying an initial request for a while. Kind of an obscure corner case but easy enough to address. And use a more convenient enum representation. * in update signals, choose `now` before acquiring the database lock. If lock acquisition takes a long time, this more accurately reflects the time the caller intended. * in general, make Time and Duration (de)serializable and use them in json types. This makes the types more self-describing, with better debug printing on both the server side and on the client library (in moonfire-playground). To make this work, base has to import serde which initially seemed like poor layering to me, but serde seems to be imported in some pretty foundational Rust crates for this reason. I'll go with it.
2021-04-21 10:44:01 -07:00
signals.times_90k.push(c.when);
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
signals.signal_ids.push(c.signal);
signals.states.push(c.state);
});
read-only signals support (#28) This is mostly untested and useless by itself, but it's a starting point. In particular: * there's no way to set up signals or add/remove/update events yet except by manual changes to the database. * if you associate a signal with a camera then remove the camera, hitting /api/ will error out.
2019-06-06 16:18:13 -07:00
serve_json(req, &signals)
}
better error handling for authentication
2021-03-06 05:16:09 -08:00
/// Authenticates the session (if any) and returns a Caller.
///
/// If there's no session,
/// 1. if `allow_unauthenticated_permissions` is configured, returns okay
/// with those permissions.
/// 2. if the caller specifies `unauth_path`, returns okay with no
/// permissions.
/// 3. returns `Unauthenticated` error otherwise.
///
/// Does no authorization. That is, this doesn't check that the returned
/// permissions are sufficient for whatever operation the caller is
/// performing.
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
fn authenticate(
&self,
req: &Request<hyper::Body>,
unauth_path: bool,
) -> Result<Caller, base::Error> {
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
if let Some(sid) = extract_sid(req) {
let authreq = self.authreq(req);
add concept of user/session permissions (I also considered the names "capabilities" and "scopes", but I think "permissions" is the most widely understood.) This is increasingly necessary as the web API becomes more capable. Among other things, it allows: * non-administrator users who can view but not access camera passwords or change any state * workers that update signal state based on cameras' built-in motion detection or a security system's events but don't need to view videos * control over what can be done without authenticating Currently session permissions are just copied from user permissions, but you can also imagine admin sessions vs not, as a checkbox when signing in. This would match the standard Unix workflow of using a non-administrative session most of the time. Relevant to my current signals work (#28) and to the addition of an administrative API (#35, including #66).
2019-06-19 15:17:50 -07:00
address some no-op clippy warnings
2021-05-17 14:31:50 -07:00
match self.db.lock().authenticate_session(authreq, &sid.hash()) {
better error handling for authentication
2021-03-06 05:16:09 -08:00
Ok((s, u)) => {
return Ok(Caller {
permissions: s.permissions.clone(),
UI preferences: #153 #155
2021-09-01 15:01:42 -07:00
user: Some(json::ToplevelUser {
id: s.user_id,
name: u.username.clone(),
add config json to user table
2021-10-26 13:08:45 -07:00
preferences: u.config.preferences.clone(),
UI preferences: #153 #155
2021-09-01 15:01:42 -07:00
session: Some(json::Session { csrf: s.csrf() }),
better error handling for authentication
2021-03-06 05:16:09 -08:00
}),
})
}
Err(e) if e.kind() == base::ErrorKind::Unauthenticated => {
// Log the specific reason this session is unauthenticated.
// Don't let the API client see it, as it may have a
// revocation reason that isn't for their eyes.
warn!("Session authentication failed: {:?}", &e);
}
Err(e) => return Err(e),
};
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
}
add concept of user/session permissions (I also considered the names "capabilities" and "scopes", but I think "permissions" is the most widely understood.) This is increasingly necessary as the web API becomes more capable. Among other things, it allows: * non-administrator users who can view but not access camera passwords or change any state * workers that update signal state based on cameras' built-in motion detection or a security system's events but don't need to view videos * control over what can be done without authenticating Currently session permissions are just copied from user permissions, but you can also imagine admin sessions vs not, as a checkbox when signing in. This would match the standard Unix workflow of using a non-administrative session most of the time. Relevant to my current signals work (#28) and to the addition of an administrative API (#35, including #66).
2019-06-19 15:17:50 -07:00
if let Some(s) = self.allow_unauthenticated_permissions.as_ref() {
return Ok(Caller {
permissions: s.clone(),
UI preferences: #153 #155
2021-09-01 15:01:42 -07:00
user: None,
add concept of user/session permissions (I also considered the names "capabilities" and "scopes", but I think "permissions" is the most widely understood.) This is increasingly necessary as the web API becomes more capable. Among other things, it allows: * non-administrator users who can view but not access camera passwords or change any state * workers that update signal state based on cameras' built-in motion detection or a security system's events but don't need to view videos * control over what can be done without authenticating Currently session permissions are just copied from user permissions, but you can also imagine admin sessions vs not, as a checkbox when signing in. This would match the standard Unix workflow of using a non-administrative session most of the time. Relevant to my current signals work (#28) and to the addition of an administrative API (#35, including #66).
2019-06-19 15:17:50 -07:00
});
}
if unauth_path {
return Ok(Caller {
permissions: db::Permissions::default(),
UI preferences: #153 #155
2021-09-01 15:01:42 -07:00
user: None,
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
});
add concept of user/session permissions (I also considered the names "capabilities" and "scopes", but I think "permissions" is the most widely understood.) This is increasingly necessary as the web API becomes more capable. Among other things, it allows: * non-administrator users who can view but not access camera passwords or change any state * workers that update signal state based on cameras' built-in motion detection or a security system's events but don't need to view videos * control over what can be done without authenticating Currently session permissions are just copied from user permissions, but you can also imagine admin sessions vs not, as a checkbox when signing in. This would match the standard Unix workflow of using a non-administrative session most of the time. Relevant to my current signals work (#28) and to the addition of an administrative API (#35, including #66).
2019-06-19 15:17:50 -07:00
}
bail_t!(Unauthenticated, "unauthenticated");
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
}
}
use SameSite=Lax instead of SameSite=Strict To improve reliability of live streams (#59) on Safari. Safari was dropping the cookie from websocket update requests. (But it worked sometimes. I don't get why.) I saw folks on the Internet thinking this related to HttpOnly: * https://developer.apple.com/forums/thread/104488 * https://stackoverflow.com/q/47742807/23584 but I still see this behavior without HttpOnly. SameSite=Strict vs SameSite=Lax appears to make a difference. Try that instead. SameSite=Strict is pointless for us anyway as noted in a new comment. Turning off HttpOnly would be more unfortunate security-wise.
2021-03-31 10:44:08 -07:00
/// Encodes a session into `Set-Cookie` header value form.
fn encode_sid(sid: db::RawSessionId, flags: i32) -> String {
let mut cookie = String::with_capacity(128);
cookie.push_str("s=");
base64::encode_config_buf(&sid, base64::STANDARD_NO_PAD, &mut cookie);
use auth::SessionFlag;
if (flags & SessionFlag::HttpOnly as i32) != 0 {
cookie.push_str("; HttpOnly");
}
if (flags & SessionFlag::Secure as i32) != 0 {
cookie.push_str("; Secure");
}
if (flags & SessionFlag::SameSiteStrict as i32) != 0 {
cookie.push_str("; SameSite=Strict");
} else if (flags & SessionFlag::SameSite as i32) != 0 {
cookie.push_str("; SameSite=Lax");
}
cookie.push_str("; Max-Age=2147483648; Path=/");
cookie
}
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 14:34:00 -08:00
#[cfg(test)]
mod tests {
split static file serving into its own file
2021-10-28 12:49:50 -07:00
use super::Segments;
lose "extern crate" everywhere (Rust 2018 edition)
2018-12-28 21:53:29 -06:00
use db::testutil::{self, TestDb};
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
use futures::future::FutureExt;
lose "extern crate" everywhere (Rust 2018 edition)
2018-12-28 21:53:29 -06:00
use log::info;
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
use std::collections::HashMap;
replace regex use with nom This reduces the binary size noticeably on my macOS machine (#70): unstripped stripped 1 before switching to clap 11.1 MiB 6.7 MiB 2 after switching to clap 11.4 MiB 6.9 MiB 3 without regex 10.1 MiB 5.9 MiB
2020-04-17 23:02:02 -07:00
use std::str::FromStr;
more readable async web code This uses "async fn" throughout rather than a mix of async and the older futures style. And it takes advantage of the "self: Arc<Self>" syntax to avoid having a ServiceInner. It was confusing to have some methods on Service and some on ServiceInner; now that distinction is gone. One downside is there's a little more atomic reference-counting. Before, service_fn essentially took an &Arc<Self>, which means it could call Arc::clone where its use of self actually outlived the future (see stream_live_m4s) but didn't need to otherwise. After, it calls an async fn that takes Arc<Self>. Using &Arc<Self> is apparently possible (as of Rust 1.41) but using that with "async fn" means the returned future is tied to its lifetime. The workaround is to use async blocks as described here: <https://rust-lang.github.io/async-book/03_async_await/01_chapter.html> but that's really ugly: it brings back the explicit Future reference, requires futures::future::Either in some cases, and introduces another level of indenting. I think it's better to just pay the arc costs which are probably negligible, or at least cheaper than the boxing was before. Oh, and I make this compile on Rust 1.40 again as it claimed to. http-serve accidentally used the &Arc<Self> thing which broke this. Update to a freshly-pushed commit which doesn't do that.
2020-05-30 19:08:54 -07:00
use std::sync::Arc;
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 14:34:00 -08:00
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
struct Server {
lose "extern crate" everywhere (Rust 2018 edition)
2018-12-28 21:53:29 -06:00
db: TestDb<base::clock::RealClocks>,
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
base_url: String,
//test_camera_uuid: Uuid,
handle: Option<::std::thread::JoinHandle<()>>,
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
shutdown_tx: Option<futures::channel::oneshot::Sender<()>>,
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
}
impl Server {
add concept of user/session permissions (I also considered the names "capabilities" and "scopes", but I think "permissions" is the most widely understood.) This is increasingly necessary as the web API becomes more capable. Among other things, it allows: * non-administrator users who can view but not access camera passwords or change any state * workers that update signal state based on cameras' built-in motion detection or a security system's events but don't need to view videos * control over what can be done without authenticating Currently session permissions are just copied from user permissions, but you can also imagine admin sessions vs not, as a checkbox when signing in. This would match the standard Unix workflow of using a non-administrative session most of the time. Relevant to my current signals work (#28) and to the addition of an administrative API (#35, including #66).
2019-06-19 15:17:50 -07:00
fn new(allow_unauthenticated_permissions: Option<db::Permissions>) -> Server {
lose "extern crate" everywhere (Rust 2018 edition)
2018-12-28 21:53:29 -06:00
let db = TestDb::new(base::clock::RealClocks {});
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
let (shutdown_tx, shutdown_rx) = futures::channel::oneshot::channel::<()>();
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
let service = Arc::new(
super::Service::new(super::Config {
db: db.db.clone(),
ui_dir: None,
allow_unauthenticated_permissions,
trust_forward_hdrs: true,
time_zone_name: "".to_owned(),
})
.unwrap(),
);
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
let make_svc = hyper::service::make_service_fn(move |_conn| {
futures::future::ok::<_, std::convert::Infallible>(hyper::service::service_fn({
more readable async web code This uses "async fn" throughout rather than a mix of async and the older futures style. And it takes advantage of the "self: Arc<Self>" syntax to avoid having a ServiceInner. It was confusing to have some methods on Service and some on ServiceInner; now that distinction is gone. One downside is there's a little more atomic reference-counting. Before, service_fn essentially took an &Arc<Self>, which means it could call Arc::clone where its use of self actually outlived the future (see stream_live_m4s) but didn't need to otherwise. After, it calls an async fn that takes Arc<Self>. Using &Arc<Self> is apparently possible (as of Rust 1.41) but using that with "async fn" means the returned future is tied to its lifetime. The workaround is to use async blocks as described here: <https://rust-lang.github.io/async-book/03_async_await/01_chapter.html> but that's really ugly: it brings back the explicit Future reference, requires futures::future::Either in some cases, and introduces another level of indenting. I think it's better to just pay the arc costs which are probably negligible, or at least cheaper than the boxing was before. Oh, and I make this compile on Rust 1.40 again as it claimed to. http-serve accidentally used the &Arc<Self> thing which broke this. Update to a freshly-pushed commit which doesn't do that.
2020-05-30 19:08:54 -07:00
let s = Arc::clone(&service);
move |req| Arc::clone(&s).serve(req)
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
}))
});
upgrade some deps, including reqwest The reqwest one is particularly notable because it means not having two versions of hyper/http/tokio/futures/bytes. It also drops a number of transitive deps; with some work I think I could stop depending on regex now.
2020-01-09 20:06:30 -08:00
let (tx, rx) = std::sync::mpsc::channel();
let handle = ::std::thread::spawn(move || {
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
let addr = ([127, 0, 0, 1], 0).into();
upgrade tokio, bytes, hyper, reqwest, http-serve This was mostly straightforward. The most confusing part waas the Sync bound change on body streams. I copied what hyper did and it seemed to work. /shruggie
2021-01-27 11:47:52 -08:00
let rt = tokio::runtime::Runtime::new().unwrap();
let srv = {
let _guard = rt.enter();
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
hyper::server::Server::bind(&addr)
.tcp_nodelay(true)
.serve(make_svc)
upgrade tokio, bytes, hyper, reqwest, http-serve This was mostly straightforward. The most confusing part waas the Sync bound change on body streams. I copied what hyper did and it seemed to work. /shruggie
2021-01-27 11:47:52 -08:00
};
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
let addr = srv.local_addr(); // resolve port 0 to a real ephemeral port number.
upgrade some deps, including reqwest The reqwest one is particularly notable because it means not having two versions of hyper/http/tokio/futures/bytes. It also drops a number of transitive deps; with some work I think I could stop depending on regex now.
2020-01-09 20:06:30 -08:00
tx.send(addr).unwrap();
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
rt.block_on(srv.with_graceful_shutdown(shutdown_rx.map(|_| ())))
.unwrap();
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
});
upgrade some deps, including reqwest The reqwest one is particularly notable because it means not having two versions of hyper/http/tokio/futures/bytes. It also drops a number of transitive deps; with some work I think I could stop depending on regex now.
2020-01-09 20:06:30 -08:00
let addr = rx.recv().unwrap();
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
// Create a user.
let mut c = db::UserChange::add_user("slamb".to_owned());
c.set_password("hunter2".to_owned());
db.db.lock().apply_user_change(c).unwrap();
Server {
db,
base_url: format!("http://{}:{}", addr.ip(), addr.port()),
handle: Some(handle),
shutdown_tx: Some(shutdown_tx),
}
}
}
impl Drop for Server {
fn drop(&mut self) {
self.shutdown_tx.take().unwrap().send(()).unwrap();
self.handle.take().unwrap().join().unwrap()
}
}
#[derive(Clone, Debug, Default)]
struct SessionCookie(Option<String>);
impl SessionCookie {
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
pub fn new(headers: &reqwest::header::HeaderMap) -> Self {
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
let mut c = SessionCookie::default();
c.update(headers);
c
}
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
pub fn update(&mut self, headers: &reqwest::header::HeaderMap) {
for set_cookie in headers.get_all(reqwest::header::SET_COOKIE) {
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
let mut set_cookie = set_cookie.to_str().unwrap().split("; ");
let c = set_cookie.next().unwrap();
let mut clear = false;
for attr in set_cookie {
if attr == "Max-Age=0" {
clear = true;
}
}
if !c.starts_with("s=") {
panic!("unrecognized cookie");
}
self.0 = if clear { None } else { Some(c.to_owned()) };
}
}
/// Produces a `Cookie` header value.
pub fn header(&self) -> String {
cleanup some awkward Option call chains
2021-05-06 09:47:07 -07:00
self.0.clone().unwrap()
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
}
}
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-20 22:08:18 -08:00
#[test]
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
#[rustfmt::skip]
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-20 22:08:18 -08:00
fn test_segments() {
testutil::init();
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
assert_eq!(
Segments { ids: 1..2, open_id: None, start_time: 0, end_time: None },
Segments::from_str("1").unwrap()
);
assert_eq!(
Segments { ids: 1..2, open_id: Some(42), start_time: 0, end_time: None },
Segments::from_str("1@42").unwrap()
);
assert_eq!(
Segments { ids: 1..2, open_id: None, start_time: 26, end_time: None },
Segments::from_str("1.26-").unwrap()
);
assert_eq!(
Segments { ids: 1..2, open_id: Some(42), start_time: 26, end_time: None },
Segments::from_str("1@42.26-").unwrap()
);
assert_eq!(
Segments { ids: 1..2, open_id: None, start_time: 0, end_time: Some(42) },
Segments::from_str("1.-42").unwrap()
);
assert_eq!(
Segments { ids: 1..2, open_id: None, start_time: 26, end_time: Some(42) },
Segments::from_str("1.26-42").unwrap()
);
assert_eq!(
Segments { ids: 1..6, open_id: None, start_time: 0, end_time: None },
Segments::from_str("1-5").unwrap()
);
assert_eq!(
Segments { ids: 1..6, open_id: None, start_time: 26, end_time: None },
Segments::from_str("1-5.26-").unwrap()
);
assert_eq!(
Segments { ids: 1..6, open_id: None, start_time: 0, end_time: Some(42) },
Segments::from_str("1-5.-42").unwrap()
);
assert_eq!(
Segments { ids: 1..6, open_id: None, start_time: 26, end_time: Some(42) },
Segments::from_str("1-5.26-42").unwrap()
);
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-20 22:08:18 -08:00
}
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
upgrade some deps, including reqwest The reqwest one is particularly notable because it means not having two versions of hyper/http/tokio/futures/bytes. It also drops a number of transitive deps; with some work I think I could stop depending on regex now.
2020-01-09 20:06:30 -08:00
#[tokio::test]
async fn unauthorized_without_cookie() {
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
testutil::init();
add concept of user/session permissions (I also considered the names "capabilities" and "scopes", but I think "permissions" is the most widely understood.) This is increasingly necessary as the web API becomes more capable. Among other things, it allows: * non-administrator users who can view but not access camera passwords or change any state * workers that update signal state based on cameras' built-in motion detection or a security system's events but don't need to view videos * control over what can be done without authenticating Currently session permissions are just copied from user permissions, but you can also imagine admin sessions vs not, as a checkbox when signing in. This would match the standard Unix workflow of using a non-administrative session most of the time. Relevant to my current signals work (#28) and to the addition of an administrative API (#35, including #66).
2019-06-19 15:17:50 -07:00
let s = Server::new(None);
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
let cli = reqwest::Client::new();
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
let resp = cli
.get(&format!("{}/api/", &s.base_url))
.send()
.await
.unwrap();
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
assert_eq!(resp.status(), reqwest::StatusCode::UNAUTHORIZED);
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
}
upgrade some deps, including reqwest The reqwest one is particularly notable because it means not having two versions of hyper/http/tokio/futures/bytes. It also drops a number of transitive deps; with some work I think I could stop depending on regex now.
2020-01-09 20:06:30 -08:00
#[tokio::test]
async fn login() {
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
testutil::init();
add concept of user/session permissions (I also considered the names "capabilities" and "scopes", but I think "permissions" is the most widely understood.) This is increasingly necessary as the web API becomes more capable. Among other things, it allows: * non-administrator users who can view but not access camera passwords or change any state * workers that update signal state based on cameras' built-in motion detection or a security system's events but don't need to view videos * control over what can be done without authenticating Currently session permissions are just copied from user permissions, but you can also imagine admin sessions vs not, as a checkbox when signing in. This would match the standard Unix workflow of using a non-administrative session most of the time. Relevant to my current signals work (#28) and to the addition of an administrative API (#35, including #66).
2019-06-19 15:17:50 -07:00
let s = Server::new(None);
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
let cli = reqwest::Client::new();
let login_url = format!("{}/api/login", &s.base_url);
upgrade some deps, including reqwest The reqwest one is particularly notable because it means not having two versions of hyper/http/tokio/futures/bytes. It also drops a number of transitive deps; with some work I think I could stop depending on regex now.
2020-01-09 20:06:30 -08:00
let resp = cli.get(&login_url).send().await.unwrap();
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
assert_eq!(resp.status(), reqwest::StatusCode::METHOD_NOT_ALLOWED);
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
upgrade some deps, including reqwest The reqwest one is particularly notable because it means not having two versions of hyper/http/tokio/futures/bytes. It also drops a number of transitive deps; with some work I think I could stop depending on regex now.
2020-01-09 20:06:30 -08:00
let resp = cli.post(&login_url).send().await.unwrap();
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
assert_eq!(resp.status(), reqwest::StatusCode::BAD_REQUEST);
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
let mut p = HashMap::new();
p.insert("username", "slamb");
p.insert("password", "asdf");
upgrade some deps, including reqwest The reqwest one is particularly notable because it means not having two versions of hyper/http/tokio/futures/bytes. It also drops a number of transitive deps; with some work I think I could stop depending on regex now.
2020-01-09 20:06:30 -08:00
let resp = cli.post(&login_url).json(&p).send().await.unwrap();
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
assert_eq!(resp.status(), reqwest::StatusCode::UNAUTHORIZED);
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
p.insert("password", "hunter2");
upgrade some deps, including reqwest The reqwest one is particularly notable because it means not having two versions of hyper/http/tokio/futures/bytes. It also drops a number of transitive deps; with some work I think I could stop depending on regex now.
2020-01-09 20:06:30 -08:00
let resp = cli.post(&login_url).json(&p).send().await.unwrap();
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
assert_eq!(resp.status(), reqwest::StatusCode::NO_CONTENT);
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
let cookie = SessionCookie::new(resp.headers());
info!("cookie: {:?}", cookie);
info!("header: {}", cookie.header());
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
let resp = cli
.get(&format!("{}/api/", &s.base_url))
.header(reqwest::header::COOKIE, cookie.header())
.send()
.await
.unwrap();
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
assert_eq!(resp.status(), reqwest::StatusCode::OK);
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
}
upgrade some deps, including reqwest The reqwest one is particularly notable because it means not having two versions of hyper/http/tokio/futures/bytes. It also drops a number of transitive deps; with some work I think I could stop depending on regex now.
2020-01-09 20:06:30 -08:00
#[tokio::test]
async fn logout() {
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
testutil::init();
add concept of user/session permissions (I also considered the names "capabilities" and "scopes", but I think "permissions" is the most widely understood.) This is increasingly necessary as the web API becomes more capable. Among other things, it allows: * non-administrator users who can view but not access camera passwords or change any state * workers that update signal state based on cameras' built-in motion detection or a security system's events but don't need to view videos * control over what can be done without authenticating Currently session permissions are just copied from user permissions, but you can also imagine admin sessions vs not, as a checkbox when signing in. This would match the standard Unix workflow of using a non-administrative session most of the time. Relevant to my current signals work (#28) and to the addition of an administrative API (#35, including #66).
2019-06-19 15:17:50 -07:00
let s = Server::new(None);
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
let cli = reqwest::Client::new();
let mut p = HashMap::new();
p.insert("username", "slamb");
p.insert("password", "hunter2");
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
let resp = cli
.post(&format!("{}/api/login", &s.base_url))
.json(&p)
.send()
.await
.unwrap();
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
assert_eq!(resp.status(), reqwest::StatusCode::NO_CONTENT);
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
let cookie = SessionCookie::new(resp.headers());
// A GET shouldn't work.
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
let resp = cli
.get(&format!("{}/api/logout", &s.base_url))
.header(reqwest::header::COOKIE, cookie.header())
.send()
.await
.unwrap();
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
assert_eq!(resp.status(), reqwest::StatusCode::METHOD_NOT_ALLOWED);
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
// Neither should a POST without a csrf token.
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
let resp = cli
.post(&format!("{}/api/logout", &s.base_url))
.header(reqwest::header::COOKIE, cookie.header())
.send()
.await
.unwrap();
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
assert_eq!(resp.status(), reqwest::StatusCode::BAD_REQUEST);
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
// But it should work with the csrf token.
// Retrieve that from the toplevel API request.
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
let toplevel: serde_json::Value = cli
.post(&format!("{}/api/", &s.base_url))
.header(reqwest::header::COOKIE, cookie.header())
.send()
.await
.unwrap()
.json()
.await
.unwrap();
let csrf = toplevel
UI preferences: #153 #155
2021-09-01 15:01:42 -07:00
.get("user")
.unwrap()
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
.get("session")
.unwrap()
.get("csrf")
.unwrap()
.as_str();
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
let mut p = HashMap::new();
p.insert("csrf", csrf);
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
let resp = cli
.post(&format!("{}/api/logout", &s.base_url))
.header(reqwest::header::COOKIE, cookie.header())
.json(&p)
.send()
.await
.unwrap();
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
assert_eq!(resp.status(), reqwest::StatusCode::NO_CONTENT);
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
let mut updated_cookie = cookie.clone();
updated_cookie.update(resp.headers());
// The cookie should be cleared client-side.
assert!(updated_cookie.0.is_none());
// It should also be invalidated server-side.
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
let resp = cli
.get(&format!("{}/api/", &s.base_url))
.header(reqwest::header::COOKIE, cookie.header())
.send()
.await
.unwrap();
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
assert_eq!(resp.status(), reqwest::StatusCode::UNAUTHORIZED);
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
}
introduce typed errors and use in mp4 code Fixes #46. If there are no video_sample_entries, it returns InvalidArgument, which gets mapped to a HTTP 400. Various other failures turn into non-500s as well. There are many places that can & should be using typed errors, but it's a start.
2018-12-28 17:30:33 -06:00
upgrade some deps, including reqwest The reqwest one is particularly notable because it means not having two versions of hyper/http/tokio/futures/bytes. It also drops a number of transitive deps; with some work I think I could stop depending on regex now.
2020-01-09 20:06:30 -08:00
#[tokio::test]
async fn view_without_segments() {
introduce typed errors and use in mp4 code Fixes #46. If there are no video_sample_entries, it returns InvalidArgument, which gets mapped to a HTTP 400. Various other failures turn into non-500s as well. There are many places that can & should be using typed errors, but it's a start.
2018-12-28 17:30:33 -06:00
testutil::init();
add concept of user/session permissions (I also considered the names "capabilities" and "scopes", but I think "permissions" is the most widely understood.) This is increasingly necessary as the web API becomes more capable. Among other things, it allows: * non-administrator users who can view but not access camera passwords or change any state * workers that update signal state based on cameras' built-in motion detection or a security system's events but don't need to view videos * control over what can be done without authenticating Currently session permissions are just copied from user permissions, but you can also imagine admin sessions vs not, as a checkbox when signing in. This would match the standard Unix workflow of using a non-administrative session most of the time. Relevant to my current signals work (#28) and to the addition of an administrative API (#35, including #66).
2019-06-19 15:17:50 -07:00
let mut permissions = db::Permissions::new();
permissions.view_video = true;
let s = Server::new(Some(permissions));
introduce typed errors and use in mp4 code Fixes #46. If there are no video_sample_entries, it returns InvalidArgument, which gets mapped to a HTTP 400. Various other failures turn into non-500s as well. There are many places that can & should be using typed errors, but it's a start.
2018-12-28 17:30:33 -06:00
let cli = reqwest::Client::new();
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
let resp = cli
.get(&format!(
"{}/api/cameras/{}/main/view.mp4",
&s.base_url, s.db.test_camera_uuid
))
.send()
.await
.unwrap();
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
assert_eq!(resp.status(), reqwest::StatusCode::BAD_REQUEST);
introduce typed errors and use in mp4 code Fixes #46. If there are no video_sample_entries, it returns InvalidArgument, which gets mapped to a HTTP 400. Various other failures turn into non-500s as well. There are many places that can & should be using typed errors, but it's a start.
2018-12-28 17:30:33 -06:00
}
use SameSite=Lax instead of SameSite=Strict To improve reliability of live streams (#59) on Safari. Safari was dropping the cookie from websocket update requests. (But it worked sometimes. I don't get why.) I saw folks on the Internet thinking this related to HttpOnly: * https://developer.apple.com/forums/thread/104488 * https://stackoverflow.com/q/47742807/23584 but I still see this behavior without HttpOnly. SameSite=Strict vs SameSite=Lax appears to make a difference. Try that instead. SameSite=Strict is pointless for us anyway as noted in a new comment. Turning off HttpOnly would be more unfortunate security-wise.
2021-03-31 10:44:08 -07:00
#[test]
fn encode_sid() {
use super::encode_sid;
use db::auth::{RawSessionId, SessionFlag};
let s64 = "3LbrruP5vj/hpE8kvYTz/rNDg4BleRiTCHGA3Ocm91z/YrtxHDxexmrz46biZJxJ";
let s = RawSessionId::decode_base64(s64.as_bytes()).unwrap();
assert_eq!(
encode_sid(
s,
(SessionFlag::Secure as i32)
| (SessionFlag::HttpOnly as i32)
| (SessionFlag::SameSite as i32)
| (SessionFlag::SameSiteStrict as i32)
),
format!(
"s={}; HttpOnly; Secure; SameSite=Strict; Max-Age=2147483648; Path=/",
s64
)
);
assert_eq!(
encode_sid(s, SessionFlag::SameSite as i32),
format!("s={}; SameSite=Lax; Max-Age=2147483648; Path=/", s64)
);
}
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 14:34:00 -08:00
}
benchmark camera page, fix broken schema This page was noticeably slower than necessary because the recording_cover index wasn't actually covering the query. Both the schema for new databases and the upgrade query were broken (and not even in the same way). No new schema version to correct this, at least for now. I'll probably have another reason to change the schema soon anyway and can throw this in.
2017-02-12 20:37:03 -08:00
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
#[cfg(all(test, feature = "nightly"))]
benchmark camera page, fix broken schema This page was noticeably slower than necessary because the recording_cover index wasn't actually covering the query. Both the schema for new databases and the upgrade query were broken (and not even in the same way). No new schema version to correct this, at least for now. I'll probably have another reason to change the schema soon anyway and can throw this in.
2017-02-12 20:37:03 -08:00
mod bench {
extern crate test;
initial split of database to a separate crate It should reduce compile time / memory usage to put quite a bit of the code into a separate crate. I also intend to limit visibility of some things to only within the db crate, but that's for a future change. This is the smallest move that will compile.
2018-02-20 23:15:39 -08:00
use db::testutil::{self, TestDb};
benchmark camera page, fix broken schema This page was noticeably slower than necessary because the recording_cover index wasn't actually covering the query. Both the schema for new databases and the upgrade query were broken (and not even in the same way). No new schema version to correct this, at least for now. I'll probably have another reason to change the schema soon anyway and can throw this in.
2017-02-12 20:37:03 -08:00
use hyper;
lose "extern crate" everywhere (Rust 2018 edition)
2018-12-28 21:53:29 -06:00
use lazy_static::lazy_static;
more readable async web code This uses "async fn" throughout rather than a mix of async and the older futures style. And it takes advantage of the "self: Arc<Self>" syntax to avoid having a ServiceInner. It was confusing to have some methods on Service and some on ServiceInner; now that distinction is gone. One downside is there's a little more atomic reference-counting. Before, service_fn essentially took an &Arc<Self>, which means it could call Arc::clone where its use of self actually outlived the future (see stream_live_m4s) but didn't need to otherwise. After, it calls an async fn that takes Arc<Self>. Using &Arc<Self> is apparently possible (as of Rust 1.41) but using that with "async fn" means the returned future is tied to its lifetime. The workaround is to use async blocks as described here: <https://rust-lang.github.io/async-book/03_async_await/01_chapter.html> but that's really ugly: it brings back the explicit Future reference, requires futures::future::Either in some cases, and introduces another level of indenting. I think it's better to just pay the arc costs which are probably negligible, or at least cheaper than the boxing was before. Oh, and I make this compile on Rust 1.40 again as it claimed to. http-serve accidentally used the &Arc<Self> thing which broke this. Update to a freshly-pushed commit which doesn't do that.
2020-05-30 19:08:54 -07:00
use std::sync::Arc;
use add_camera in tests, not direct db inserts This is a wash in terms of lines of code now, but it makes it a bit easier to maintain as I make changes to the schema (such as separating out streams from cameras), and it helps ensure the tests reflect reality.
2018-02-03 21:56:04 -08:00
use uuid::Uuid;
benchmark camera page, fix broken schema This page was noticeably slower than necessary because the recording_cover index wasn't actually covering the query. Both the schema for new databases and the upgrade query were broken (and not even in the same way). No new schema version to correct this, at least for now. I'll probably have another reason to change the schema soon anyway and can throw this in.
2017-02-12 20:37:03 -08:00
struct Server {
base_url: String,
use add_camera in tests, not direct db inserts This is a wash in terms of lines of code now, but it makes it a bit easier to maintain as I make changes to the schema (such as separating out streams from cameras), and it helps ensure the tests reflect reality.
2018-02-03 21:56:04 -08:00
test_camera_uuid: Uuid,
benchmark camera page, fix broken schema This page was noticeably slower than necessary because the recording_cover index wasn't actually covering the query. Both the schema for new databases and the upgrade query were broken (and not even in the same way). No new schema version to correct this, at least for now. I'll probably have another reason to change the schema soon anyway and can throw this in.
2017-02-12 20:37:03 -08:00
}
impl Server {
fn new() -> Server {
refine flush_if_sec behavior The new behavior eliminates a couple unpleasant edge cases in which it would never flush: * if all recording stops, whatever was unflushed would stay that way * if every recording attempt produces a 0-duration recording (such as if the camera sends only one frame and thus no PTS delta can be calculated), the list of recordings to flush would continue to grow
2018-03-23 15:16:43 -07:00
let db = TestDb::new(::base::clock::RealClocks {});
use add_camera in tests, not direct db inserts This is a wash in terms of lines of code now, but it makes it a bit easier to maintain as I make changes to the schema (such as separating out streams from cameras), and it helps ensure the tests reflect reality.
2018-02-03 21:56:04 -08:00
let test_camera_uuid = db.test_camera_uuid;
benchmark camera page, fix broken schema This page was noticeably slower than necessary because the recording_cover index wasn't actually covering the query. Both the schema for new databases and the upgrade query were broken (and not even in the same way). No new schema version to correct this, at least for now. I'll probably have another reason to change the schema soon anyway and can throw this in.
2017-02-12 20:37:03 -08:00
testutil::add_dummy_recordings_to_db(&db.db, 1440);
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
let service = Arc::new(
super::Service::new(super::Config {
db: db.db.clone(),
ui_dir: None,
allow_unauthenticated_permissions: Some(db::Permissions::default()),
trust_forward_hdrs: false,
time_zone_name: "".to_owned(),
})
.unwrap(),
);
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
let make_svc = hyper::service::make_service_fn(move |_conn| {
futures::future::ok::<_, std::convert::Infallible>(hyper::service::service_fn({
more readable async web code This uses "async fn" throughout rather than a mix of async and the older futures style. And it takes advantage of the "self: Arc<Self>" syntax to avoid having a ServiceInner. It was confusing to have some methods on Service and some on ServiceInner; now that distinction is gone. One downside is there's a little more atomic reference-counting. Before, service_fn essentially took an &Arc<Self>, which means it could call Arc::clone where its use of self actually outlived the future (see stream_live_m4s) but didn't need to otherwise. After, it calls an async fn that takes Arc<Self>. Using &Arc<Self> is apparently possible (as of Rust 1.41) but using that with "async fn" means the returned future is tied to its lifetime. The workaround is to use async blocks as described here: <https://rust-lang.github.io/async-book/03_async_await/01_chapter.html> but that's really ugly: it brings back the explicit Future reference, requires futures::future::Either in some cases, and introduces another level of indenting. I think it's better to just pay the arc costs which are probably negligible, or at least cheaper than the boxing was before. Oh, and I make this compile on Rust 1.40 again as it claimed to. http-serve accidentally used the &Arc<Self> thing which broke this. Update to a freshly-pushed commit which doesn't do that.
2020-05-30 19:08:54 -07:00
let s = Arc::clone(&service);
move |req| Arc::clone(&s).serve(req)
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
}))
});
upgrade tokio, bytes, hyper, reqwest, http-serve This was mostly straightforward. The most confusing part waas the Sync bound change on body streams. I copied what hyper did and it seemed to work. /shruggie
2021-01-27 11:47:52 -08:00
let rt = tokio::runtime::Runtime::new().unwrap();
let srv = {
let _guard = rt.enter();
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
let addr = ([127, 0, 0, 1], 0).into();
hyper::server::Server::bind(&addr)
.tcp_nodelay(true)
.serve(make_svc)
upgrade tokio, bytes, hyper, reqwest, http-serve This was mostly straightforward. The most confusing part waas the Sync bound change on body streams. I copied what hyper did and it seemed to work. /shruggie
2021-01-27 11:47:52 -08:00
};
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
let addr = srv.local_addr(); // resolve port 0 to a real ephemeral port number.
benchmark camera page, fix broken schema This page was noticeably slower than necessary because the recording_cover index wasn't actually covering the query. Both the schema for new databases and the upgrade query were broken (and not even in the same way). No new schema version to correct this, at least for now. I'll probably have another reason to change the schema soon anyway and can throw this in.
2017-02-12 20:37:03 -08:00
::std::thread::spawn(move || {
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
rt.block_on(srv).unwrap();
benchmark camera page, fix broken schema This page was noticeably slower than necessary because the recording_cover index wasn't actually covering the query. Both the schema for new databases and the upgrade query were broken (and not even in the same way). No new schema version to correct this, at least for now. I'll probably have another reason to change the schema soon anyway and can throw this in.
2017-02-12 20:37:03 -08:00
});
use add_camera in tests, not direct db inserts This is a wash in terms of lines of code now, but it makes it a bit easier to maintain as I make changes to the schema (such as separating out streams from cameras), and it helps ensure the tests reflect reality.
2018-02-03 21:56:04 -08:00
Server {
base_url: format!("http://{}:{}", addr.ip(), addr.port()),
test_camera_uuid,
}
benchmark camera page, fix broken schema This page was noticeably slower than necessary because the recording_cover index wasn't actually covering the query. Both the schema for new databases and the upgrade query were broken (and not even in the same way). No new schema version to correct this, at least for now. I'll probably have another reason to change the schema soon anyway and can throw this in.
2017-02-12 20:37:03 -08:00
}
}
lazy_static! {
live stream frame-by-frame rather than GOP-by-GOP (#59) This should reduce live stream latency by two seconds when my cameras are at their default setting (I frame interval = 2 * frame rate)! I was under the impression that every HTML5 Media Source Extensions media segment had to start with a Random Access Point. This used to be true, but apparently changed quite a while ago: https://bugs.chromium.org/p/chromium/issues/detail?id=229412 Support generating segments that don't start with a key frame, and plumb this through the mp4 media segment generation logic. Add some extra error checking in mp4 slice handling, as my first attempts had a mismatch between expected and actual lengths that silently returned corrupted .m4s files. Also pull everything from the most recent key frame on along with the first live segment to reduce startup latency. Live view is quite a bit more pleasant now.
2020-08-07 15:30:22 -07:00
static ref SERVER: Server = Server::new();
benchmark camera page, fix broken schema This page was noticeably slower than necessary because the recording_cover index wasn't actually covering the query. Both the schema for new databases and the upgrade query were broken (and not even in the same way). No new schema version to correct this, at least for now. I'll probably have another reason to change the schema soon anyway and can throw this in.
2017-02-12 20:37:03 -08:00
}
#[bench]
lose "extern crate" everywhere (Rust 2018 edition)
2018-12-28 21:53:29 -06:00
fn serve_stream_recordings(b: &mut test::Bencher) {
benchmark camera page, fix broken schema This page was noticeably slower than necessary because the recording_cover index wasn't actually covering the query. Both the schema for new databases and the upgrade query were broken (and not even in the same way). No new schema version to correct this, at least for now. I'll probably have another reason to change the schema soon anyway and can throw this in.
2017-02-12 20:37:03 -08:00
testutil::init();
let server = &*SERVER;
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
let url = reqwest::Url::parse(&format!(
"{}/api/cameras/{}/main/recordings",
server.base_url, server.test_camera_uuid
))
.unwrap();
update dependencies
2017-11-16 23:01:09 -08:00
let client = reqwest::Client::new();
upgrade tokio, bytes, hyper, reqwest, http-serve This was mostly straightforward. The most confusing part waas the Sync bound change on body streams. I copied what hyper did and it seemed to work. /shruggie
2021-01-27 11:47:52 -08:00
let rt = tokio::runtime::Runtime::new().unwrap();
let f = || {
fix some reqwest upgrade brokenness in benches
2020-02-16 23:58:07 -08:00
rt.block_on(async {
let resp = client.get(url.clone()).send().await.unwrap();
assert_eq!(resp.status(), reqwest::StatusCode::OK);
let _b = resp.bytes().await.unwrap();
});
reuse reqwest client in serve_camera_html bench This makes a huge difference in the reported time - 863 usec rather than 6 milliseconds on my laptop. Part of the difference is in reqwest client setup (it apparently initializes a SSL_CTX that is never used in this test), part fresh connections vs keepalive, part I don't know what. None of it seems relevant to the logic I want to test.
2017-03-03 22:26:29 -08:00
};
rustfmt everything I want to make the project more accessible by not expecting folks to match my idiosyncratic style. Now almost [1] everything is written in the "standard" style. CI enforces this. [1] "Almost": I used #[rustfmt::skip] in a few sections where I felt aligning things in columns significantly improves readability.
2021-02-16 22:15:54 -08:00
f(); // warm.
reuse reqwest client in serve_camera_html bench This makes a huge difference in the reported time - 863 usec rather than 6 milliseconds on my laptop. Part of the difference is in reqwest client setup (it apparently initializes a SSL_CTX that is never used in this test), part fresh connections vs keepalive, part I don't know what. None of it seems relevant to the logic I want to test.
2017-03-03 22:26:29 -08:00
b.iter(f);
benchmark camera page, fix broken schema This page was noticeably slower than necessary because the recording_cover index wasn't actually covering the query. Both the schema for new databases and the upgrade query were broken (and not even in the same way). No new schema version to correct this, at least for now. I'll probably have another reason to change the schema soon anyway and can throw this in.
2017-02-12 20:37:03 -08:00
}
}
Reference in New Issue Copy Permalink