* In 0866b239, while fixing a clippy error, I accidentally inverted the
error condition.
* While I'm at it, improve the diagnostics. Print which field we're
talking about and the expected URL schemes.
This stops using parking_lot entirely. Since Rust 1.62, the std
implementations on Linux are direct futexes, not the boxed pthread
mutexes they used to be. No real reason to use parking_lot anymore, so
shed this dependency.
Fixes#206. 307a388 switched to creating a single-threaded runtime for
each stream, then destroying prior to waiting for TEARDOWN on shutdown.
This meant that the shutdown process could panic with this error:
```
panic at '/home/slamb/git/retina/src/client/mod.rs:219:22': teardown Sender shouldn't be dropped: RecvError(())
```
Let's switch back to expecting a multithreaded runtime context.
Create one for the config subcommand, too.
Don't go all the way back to the old code with its channels, though.
That had the downside that the underlying retina::Session might outlive
the caller, so there could still be an active session when we start
the next one. I haven't seen this cause problems in practice but it
still doesn't seem right.
This alone improves interop and diagnostics, as noted in Retina's
release notes. We also now give the camera name to the session group
(for improved logging of TEARDOWN operations) and expose the RTSP
server's "tool" attribute in debug logs and the config UI's "Test"
button.
Fixes#209Fixes#213
This fixes a real cross-site WebSocket hijacking (CSWSH) vulnerability.
If the attacker knows the URL of an NVR installation this user is
authenticated to and the UUID of a camera, and can trick the user into
visiting their webpage, they can grab the live stream. At least there's
some entropy in the camera UUID, but it was never intended to be a
secret.
* switch the config interface over to use Retina and make the test
button honor rtsp_transport = udp.
* adjust the threading model of the Retina streaming code.
Before, it spawned a background future that read from the runtime and
wrote to a channel. Other calls read from this channel.
After, it does work directly from within the block_on calls (no
channels).
The immediate motivation was that the config interface didn't have
another runtime handy. And passing in a current thread runtime
deadlocked. I later learned this is a difference between
Runtime::block_on and Handle::block_on. The former will drive IO and
timers; the latter will not.
But this is also more efficient to avoid so many thread hand-offs.
Both the context switches and the extra spinning that
tokio appears to do as mentioned here:
https://github.com/scottlamb/retina/issues/5#issuecomment-871971550
This may not be the final word on the threading model. Eventually
I may not have per-stream writing threads at all. But I think it will
be easier to look at this after getting rid of the separate
`moonfire-nvr config` subcommand in favor of a web interface.
* in tests, read `.mp4` files via the `mp4` crate rather than ffmpeg.
The annoying part is that this doesn't parse edit lists; oh well.
* simplify the `Opener` interface. Formerly, it'd take either a RTSP
URL or a path to a `.mp4` file, and they'd share some code because
they both sometimes used ffmpeg. Now, they're totally different
libraries (`retina` vs `mp4`). Pull the latter out to a `testutil`
module with a different interface that exposes more of the `mp4`
stuff. Now `Opener` is just for RTSP.
* simplify the h264 module. It had a lot of logic to deal with Annex B.
Retina doesn't use this encoding.
Fixes#36Fixes#126
* switch from json to toml.
I think this will be more user-friendly. It allows comments and has
less punctuation. Fewer surprises than yaml (which has e.g. the
"Norway problem"). I might have stayed with JSON if I could see a
good serde json library that allows comments, but hson is unmaintained
and serde-json strictly follows the spec.
* switch from camelCase to snake_case. Seems more idiomatic for TOML
and matches the Rust source.
* forbid unknown keys. Better to spot errors sooner.
* rename "trust_forward_hdrs" to "trust_forward_headers". Nothing else
is abbreviated.
This is better particularly when the user is following the docker
instructions and doesn't have a local checkout at all. It also is a
rendered HTML view rather than raw markdown.
It'd be nice to link to the exact release we're using, not tip of
master. I didn't do this now because it'll likely take some work with
build.rs to check if the user is on a tagged release or not.
Fixes#180
Since 0.7.0, the one in the db package is used both for actual storage
within the database and for API use. I left the API-specific version
around by accident.
Now there's room to add arbitrary configuration to signals and types.
Several things are no longer fixed columns/tables but instead within
the configuration types.
This fixes#178. Before, everything got translated to 5xx status;
now it produces the correct type in several cases.
Ideally I'd get rid of the untyped errors in all of web.rs; this is
a small step.
Before it would produce this incorrect message that told you to run
the command you just ran:
```
$ nvr init --db-dir=/nonexistent/db
E20211021 09:08:23.798 main moonfire_nvr] Exiting due to error: db dir /nonexistent/db not found; try running moonfire-nvr init
caused by: ENOENT: No such file or directory
```
Now the same command produces the following:
```
$ nvr init --db-dir=/nonexistent/db
E20211021 09:09:11.056 main moonfire_nvr] Exiting due to error: unable to create db dir /nonexistent/db
caused by: ENOENT: No such file or directory
```
Add tests just for good measure.
After a frustrating search for a suitable channel to use for shutdown
(tokio::sync:⌚:Receiver and
futures::future::Shared<tokio::sync::oneshot::Receiver> didn't look
quite right) in which I rethought my life decisions, I finally just made
my own (server/base/shutdown.rs). We can easily poll it or wait for it
in async or sync contexts. Most importantly, it's convenient; not that
it really matters here, but it's also efficient.
We now do a slightly better job of propagating a "graceful" shutdown
signal, and this channel will give us tools to improve it over time.
* Shut down even when writer or syncer operations are stuck. Fixes#117
* Not done yet: streamers should instantly shut down without waiting for
a connection attempt or frame or something. I'll probably
implement that when removing --rtsp-library=ffmpeg. The code should be
cleaner then.
* Not done yet: fix a couple places that sleep for up to a second when
they could shut down immediately. I just need to do the plumbing for
mock clocks to work.
I also implemented an immediate shutdown mode, activated by a second
signal. I think this will mitigate the streamer wait situation.
* upgrade to Retina 0.3.1 which automatically tears down sessions
* wait out stale sessions before reconnecting
* wait for teardown to complete before shutting down
This adds some pressure on #117: it will keep waiting for the stale
session to expire even if the user has requested shutdown. I'll try
to address that next.
I copied the example of the password field by introducing a setter.
But I forgot: it was only that way because the password field has
the complexity of hashing/salting. For fields where setting is
idempotent, it can be directly exposed.
I forgot to do this in 27395ec resulting in #146. Bump the version now,
and update the digest used in the tests so they will remind me to
bump the version any time the headers change.
As written in the changelog: Live streams formerly worked around a
Firefox pixel aspect ratio bug by forcing all videos to 16:9, which
dramatically distorted 9:16 camera views. Playback didn't, so anamorphic
videos looked correct on Chrome but slightly stretched on Firefox. Now
both live streams and playback are fully correct on all browsers.
Fixes#136
Before:
```
E20210803 09:00:31.161 main moonfire_nvr] panic at '/Users/slamb/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.14.10/src/server/server.rs:68:17': error binding to 0.0.0.0:80: error creating server listener: Address already in use (os error 48)
(set environment variable RUST_BACKTRACE=1 to see backtraces)
...potentially unrelated log msgs from other threads before exiting...
```
After:
```
E20210803 09:06:02.633 main moonfire_nvr] Exiting due to error: unable to bind --http-addr=0.0.0.0:80
caused by: error creating server listener: Address already in use (os error 48)
(set environment variable RUST_BACKTRACE=1 to see backtraces)
```
For #136. I'm also going to make this particular case no longer panic,
but there will surely be other affected panics.
Before:
```
E20210803 08:58:31.606 main moonfire_nvr] panic at '/Users/slamb/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.14.10/src/server/server.rs:68:17'
```
After:
```
E20210803 08:59:51.319 main moonfire_nvr] panic at '/Users/slamb/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.14.10/src/server/server.rs:68:17': error binding to 0.0.0.0:80: error creating server listener: Address already in use (os error 48)
```
While I'm here, return a clean error if a non-initial video frame
includes a parameter change, rather than doing something crazy (#42).
It's still broken under ffmpeg, it's untested, and it's not as clean
as seamlessly starting a new recording with the new parameters, but
it's better than nothing.
* have a timeout for opening the connection and getting the next
video frame. The former is quite important. The latter is arguably
redundant with the keepalive timer, but this ensures we actually
get a full frame in this timespan rather than some keepalive
responses, RTCP sender reports, or partial frames.
* don't drop extra stuff on loss; just note it. I'm not sure what the
right behavior is but I think I shouldn't change too much at once.
I see a lot of yields and such in CPU profiles. I think the workers
are frequently waking up, finding there's not much to do, and going back
to sleep. Reducing the number of worker threads seems reasonable.
Moonfire NVR has some enforcement on its own; this makes retina vs
ffmpeg more of an apples-to-apples comparison.
I'm also thinking of dropping enforcement from retina; enough things
have sketchy timestamps that this policy doesn't make much sense anyway.
This isn't well-tested and doesn't yet support an initial connection
timeout. But in a quick test, it successfully returns video!
I'd like to do some more aggressive code restructuring for zero-copy
and to have only one writer thread per sample file directory (rather
than the syncer thread + one writer thread per RTSP stream). But I'll
likely wait until I drop support for ffmpeg entirely.
This is (slightly) complicating the switch from ffmpeg to retina
as the RTSP client. And it's not really that close to what I want
to end up with for analytics:
* I'd prefer the analytics happen in a separate process for
several reasons
* Feeding the entire frame to the object detector doesn't produce
good results.
* It doesn't do anything with the results yet anyway.
This caused served chunks to be truncated. On seek, nginx sometimes
served 502 errors, chrome sometimes returned
ERR_CONTENT_LENGTH_MISMATCH, and videos weren't playing properly.
Reading from the mmap()ed region in the tokio threads could cause
them to stall:
* That could affect UI serving when there were concurrent
UI requests (i.e., not just requests that needed the reads in
question anyway).
* If there's a faulty disk, it could cause the UI to totally hang.
Better to not mix disks between threads.
* Soon, I want to handle RTSP from the tokio threads (#37). Similarly,
we don't want RTSP streaming to block on operations from unrelated
disks.
I went with just one thread per disk which I think is sufficient.
But it'd be possible to do a fixed-size pool instead which might improve
latency when some pages are already cached.
I also dropped the memmap dependency. I had to compute the page
alignment anyway to get mremap to work, and Moonfire NVR already is
Unix-specific, so there wasn't much value from the memmap or memmap2
crates.
Fixes#88
* my dad's GW4089IP cameras use 720x480
* some Reolink cameras use 640x352
* I'm playing with rotated cameras (16x9 -> 9x16)
I'd prefer to calculate pasp from a configured camera aspect ratio
than to hardcode the assumption these are 16x9, but that requires
a schema change. This is an improvement for now.
* API change: in update signals, allow setting a start time relative
to now. This is an accuracy improvement in the case where the client
has been retrying an initial request for a while. Kind of an obscure
corner case but easy enough to address. And use a more convenient
enum representation.
* in update signals, choose `now` before acquiring the database lock.
If lock acquisition takes a long time, this more accurately reflects
the time the caller intended.
* in general, make Time and Duration (de)serializable and use them
in json types. This makes the types more self-describing, with
better debug printing on both the server side and on the client
library (in moonfire-playground). To make this work, base has to
import serde which initially seemed like poor layering to me, but
serde seems to be imported in some pretty foundational Rust crates
for this reason. I'll go with it.
In particular, this was happening out of the box on Raspberry Pi OS Lite
20210304, as reported by ironoxidizer@gmail.com here:
https://groups.google.com/g/moonfire-nvr-users/c/2j9LvfFl2u8/m/tJcNS2WfCQAJ
* adjust main.rs to make the problem more obvious
* mention it in the troubleshooting guide
* sidestep it in the nvr docker wrapper script
also just use --networking=host rather than --publish (avoiding a proxy
process). I'm using Docker to simplify the build and deployment process,
not as a security boundary, so just do the simpler thing.
To improve reliability of live streams (#59) on Safari.
Safari was dropping the cookie from websocket update requests.
(But it worked sometimes. I don't get why.) I saw folks on the Internet
thinking this related to HttpOnly:
* https://developer.apple.com/forums/thread/104488
* https://stackoverflow.com/q/47742807/23584
but I still see this behavior without HttpOnly. SameSite=Strict vs
SameSite=Lax appears to make a difference. Try that instead.
SameSite=Strict is pointless for us anyway as noted in a new comment.
Turning off HttpOnly would be more unfortunate security-wise.
As required for live view (#59) to work on Safari.
Safari has some "interesting" expectations:
* There must be a non-empty list of compatible brands. The major brand
is not automatically included. (Looks like ISO/IEC 14496-12 doesn't
spell out which is correct.)
* The tfdt box must be before the trun boxes. Moonfire NVR was not
compliant with ISO/IEC 14496-12:2015 section 8.8.12.1 before.
Chrome and Firefox didn't care, but Safari does.
* The mdat must be written with the small format. Safari is not
implementing the spec properly.
I figured these out by painstakingly comparing Moonfire NVR's output
with gpac's, making it match almost byte-for-byte until it worked, then
backing out changes one at a time to check which were relevant. Ugh!
Chrome appears to time out at 60 seconds of inactivity otherwise.
I think it's better to keep the stream open, even if the camera is
broken.
The implementation looks awkward, but that might be the state of Rust
async right now.
The CI nightly builds had been broken with the following error:
```
error: custom inner attributes are unstable
--> /home/runner/work/moonfire-nvr/moonfire-nvr/server/target/debug/build/moonfire-db-415ce696a754c614/out/schema.rs:10:4
|
10 | #![rustfmt::skip]
| ^^^^^^^^^^^^^
|
= note: `#[deny(soft_unstable)]` on by default
= warning: this was previously accepted by the compiler but is being phased out; it will become a hard error in a future release!
= note: for more information, see issue #64266 <https://github.com/rust-lang/rust/issues/64266>
```
I'd thought this was by mistake given that #[rustfmt::skip] is still
advertised on rustfmt's github page, but maybe not. Looks like
rust-protobuf's newest version uses
`#![cfg_attr(rustfmt, rustfmt::skip)]` to avoid this error.
Also fix a warning on nightly about an extraneous semicolon.
I also enforced some invariants in the signals code, fixing a couple
bugs. The signals code is more complex than I'd like, but hopefully
is working now.
My main goal is to support creating indexes for signals as well as
recordings. An additional goal is to just shrink db.rs a bit; it's
gotten quite large.
I think this has a minor behavior change: permission denied replies
change to HTTP 403 where they were HTTP 401. The new behavior seems
more correct, as these errors can occur when authentication has
succeeded but the session in question is not authorized for the given
operation. The UI currently doesn't care about this distinction.
