MyFavMirrors
/
moonfire-nvr
mirror of https://github.com/scottlamb/moonfire-nvr.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
moonfire-nvr/server/Cargo.toml

88 lines
2.5 KiB
TOML
Raw Normal View History

Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
[package]
name = "moonfire-nvr"
prepare 0.7.5 with Retina 0.3.10 Fixes #224
2022-05-09 16:08:52 -04:00
version = "0.7.5"
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
authors = ["Scott Lamb <slamb@slamb.org>"]
lose "extern crate" everywhere (Rust 2018 edition)
2018-12-28 22:53:29 -05:00
edition = "2018"
use newer cargo version resolver This trims ~700KiB off moonfire-nvr's text section, much of it by eliminating h2. See #70.
2021-10-27 16:18:52 -04:00
resolver = "2"
shorten per-file copyright headers I'm tired of all the boilerplate, so use the new GPL-3.0-linking-exception license identifier instead in all the server components. I left the ui stuff alone because I'm just going to replace it (#111). Add a checker for the header because it's easy to forget.
2021-02-17 16:28:48 -05:00
license-file = "../LICENSE.txt"
update a few deps
2022-05-02 14:22:14 -04:00
rust-version = "1.60"
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
compile with stable Rust The benchmarks now require "cargo bench --features=nightly". The extra #[cfg(nightly)] switches in the code needed for it are a bit annoying; I may move the benches to a separate directory to avoid this. But for now, this works.
2016-12-10 01:04:35 -05:00
[features]
upgrade to async hyper serve_generated_bytes is >3X faster. One caveat is that the reactor thread may stall when reading from the memory-mapped slice. Moonfire NVR is basically a single-user program, so that may not be so bad, but we'll see.
2017-03-02 22:29:28 -05:00
# The nightly feature is used within moonfire-nvr itself to gate the
# benchmarks. Also pass it along to crates that can benefit from it.
always use smallvec unions They only require Rust 1.51, and our minimum is 1.52 now.
2021-10-27 11:05:07 -04:00
nightly = ["db/nightly", "parking_lot/nightly"]
compile with stable Rust The benchmarks now require "cargo bench --features=nightly". The extra #[cfg(nightly)] switches in the code needed for it are a bit annoying; I may move the benches to a separate directory to avoid this. But for now, this works.
2016-12-10 01:04:35 -05:00
upgrade rusqlite and support bundled library SQLite3 has gotten some noticeable speed improvements in recent releases. This is a modest improvement on reasonably new platforms, and a pretty significant one on Raspbian Jessie (which has an incredibly old SQLite3).
2017-03-04 01:42:13 -05:00
# The bundled feature includes bundled (aka statically linked) versions of
# native libraries where possible.
bundled = ["rusqlite/bundled"]
initial split of database to a separate crate It should reduce compile time / memory usage to put quite a bit of the code into a separate crate. I also intend to limit visibility of some things to only within the db crate, but that's for a future change. This is the smallest move that will compile.
2018-02-21 02:15:39 -05:00
[workspace]
use extracted ffmpeg library
2020-03-28 03:59:25 -04:00
members = ["base", "db"]
initial split of database to a separate crate It should reduce compile time / memory usage to put quite a bit of the code into a separate crate. I also intend to limit visibility of some things to only within the db crate, but that's for a future change. This is the smallest move that will compile.
2018-02-21 02:15:39 -05:00
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
[dependencies]
lose "extern crate" everywhere (Rust 2018 edition)
2018-12-28 22:53:29 -05:00
base = { package = "moonfire-base", path = "base" }
update various deps This brings most things reasonably up-to-date. libpasta's deps are dragging a bit, keeping us on an older ring to avoid duplication, and causing us to use three versions of base64. And I need to update a few of my companion crates' parking_lot dep to match tokio.
2020-11-23 03:23:03 -05:00
base64 = "0.13.0"
bump versions of blake3, nom, tokio-tungstenite
2021-08-31 16:04:26 -04:00
blake3 = "1.0.0"
upgrade tokio, bytes, hyper, reqwest, http-serve This was mostly straightforward. The most confusing part waas the Sync bound change on body streams. I copied what hyper did and it seemed to work. /shruggie
2021-01-27 14:47:52 -05:00
bytes = "1"
update several dependencies I left serde alone because uuid hasn't been updated for the new version.
2017-01-27 23:58:04 -05:00
byteorder = "1.0"
tweak deps, eliminating strsim 0.8 dependency I also enabled the colored help option for clap, since we're paying for the color dep anyway.
2021-06-15 03:45:11 -04:00
clap = { version = "2.33.3", default-features = false, features = ["color", "wrap_help"] }
upgrade cursive
2022-03-08 14:35:29 -05:00
cursive = "0.17.0"
lose "extern crate" everywhere (Rust 2018 edition)
2018-12-28 22:53:29 -05:00
db = { package = "moonfire-db", path = "db" }
replace homegrown Error with failure crate This reduces boilerplate, making it a bit easier for me to split the db stuff out into its own crate.
2018-02-21 01:46:14 -05:00
failure = "0.1.1"
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-09 02:04:36 -05:00
futures = "0.3"
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
fnv = "1.0"
use released versions of a few deps
2021-06-09 17:36:14 -04:00
h264-reader = "0.5.0"
upgrade tokio, bytes, hyper, reqwest, http-serve This was mostly straightforward. The most confusing part waas the Sync bound change on body streams. I copied what hyper did and it seemed to work. /shruggie
2021-01-27 14:47:52 -05:00
http = "0.2.3"
read sample files from dedicated threads Reading from the mmap()ed region in the tokio threads could cause them to stall: * That could affect UI serving when there were concurrent UI requests (i.e., not just requests that needed the reads in question anyway). * If there's a faulty disk, it could cause the UI to totally hang. Better to not mix disks between threads. * Soon, I want to handle RTSP from the tokio threads (#37). Similarly, we don't want RTSP streaming to block on operations from unrelated disks. I went with just one thread per disk which I think is sufficient. But it'd be possible to do a fixed-size pool instead which might improve latency when some pages are already cached. I also dropped the memmap dependency. I had to compute the page alignment anyway to get mremap to work, and Moonfire NVR already is Unix-specific, so there wasn't much value from the memmap or memmap2 crates. Fixes #88
2021-06-04 22:17:32 -04:00
http-serve = { version = "0.3.1", features = ["dir"] }
upgrade tokio, bytes, hyper, reqwest, http-serve This was mostly straightforward. The most confusing part waas the Sync bound change on body streams. I copied what hyper did and it seemed to work. /shruggie
2021-01-27 14:47:52 -05:00
hyper = { version = "0.14.2", features = ["http1", "server", "stream", "tcp"] }
upgrade deps
2018-01-31 01:05:39 -05:00
lazy_static = "1.0"
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
libc = "0.2"
logging improvements * allow debug/trace logging on release builds again * enable log messages from hyper. I didn't notice they went away with 0.14.0, although there's a breaking change in the log: https://github.com/hyperium/hyper/blob/master/CHANGELOG.md#v0140-2020-12-23 * downgrade some particularly spammy messages
2021-06-05 02:28:48 -04:00
log = { version = "0.4" }
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-26 00:31:50 -05:00
memchr = "2.0.2"
specify git path to mylog
2017-03-26 03:03:24 -04:00
mylog = { git = "https://github.com/scottlamb/mylog" }
update various Rust deps
2021-09-29 14:44:29 -04:00
nix = "0.23.0"
bump versions of blake3, nom, tokio-tungstenite
2021-08-31 16:04:26 -04:00
nom = "7.0.0"
fix #187 via a dependency upgrade I did a full `cargo upgrade` and fixed what it broke: * a couple things for the latest protobuf 3.0 alphas (note alphas don't promise API stability) * new minimum supported Rust version This should have some other nice effects: parking_lot now uses inline assembler, tokio has gotten faster, etc.
2022-03-08 14:22:01 -05:00
parking_lot = "0.12.0"
switch from libpasta to just scrypt This drops several older dependencies and reduces final binary size (text section by ~200KiB, unstripped binary by ~12MiB) I'll have to manually add new hash formats, and I won't ever be able to take advantage of libpasta's (currently unused) facility to wrap hashes, but I think it's worth it. libpasta isn't well-maintained.
2021-10-27 14:38:56 -04:00
password-hash = "0.3.2"
update a few deps
2022-05-02 14:22:14 -04:00
protobuf = "3.0"
upgrade reffers dependency
2022-03-08 14:41:00 -05:00
reffers = "0.7.0"
upgrade to Retina 0.4.0
2022-05-17 19:36:39 -04:00
retina = "0.4.0"
update libpasta/ring This uses my (hopefully short-lived) libpasta fork.
2020-11-24 00:47:02 -05:00
ring = "0.16.2"
upgrade rusqlite
2022-03-08 14:37:45 -05:00
rusqlite = "0.27.0"
lose "extern crate" everywhere (Rust 2018 edition)
2018-12-28 22:53:29 -05:00
serde = { version = "1.0", features = ["derive"] }
update dependencies * The mylog update fixes a couple bad bugs. * Otherwise, just keep up with the Rust ecosystem.
2017-06-11 15:57:55 -04:00
serde_json = "1.0"
always use smallvec unions They only require Rust 1.51, and our minimum is 1.52 now.
2021-10-27 11:05:07 -04:00
smallvec = { version = "1.7", features = ["union"] }
tweak deps, eliminating strsim 0.8 dependency I also enabled the colored help option for clap, since we're paying for the color dep anyway.
2021-06-15 03:45:11 -04:00
structopt = { version = "0.3.13", default-features = false }
upgrade tokio, bytes, hyper, reqwest, http-serve This was mostly straightforward. The most confusing part waas the Sync bound change on body streams. I copied what hyper did and it seemed to work. /shruggie
2021-01-27 14:47:52 -05:00
sync_wrapper = "0.1.0"
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
time = "0.1"
shutdown better After a frustrating search for a suitable channel to use for shutdown (tokio::sync::watch::Receiver and futures::future::Shared<tokio::sync::oneshot::Receiver> didn't look quite right) in which I rethought my life decisions, I finally just made my own (server/base/shutdown.rs). We can easily poll it or wait for it in async or sync contexts. Most importantly, it's convenient; not that it really matters here, but it's also efficient. We now do a slightly better job of propagating a "graceful" shutdown signal, and this channel will give us tools to improve it over time. * Shut down even when writer or syncer operations are stuck. Fixes #117 * Not done yet: streamers should instantly shut down without waiting for a connection attempt or frame or something. I'll probably implement that when removing --rtsp-library=ffmpeg. The code should be cleaner then. * Not done yet: fix a couple places that sleep for up to a second when they could shut down immediately. I just need to do the plumbing for mock clocks to work. I also implemented an immediate shutdown mode, activated by a second signal. I think this will mitigate the streamer wait situation.
2021-09-23 18:55:53 -04:00
tokio = { version = "1.0", features = ["macros", "parking_lot", "rt-multi-thread", "signal", "sync", "time"] }
send keepalives on live.m4s Chrome appears to time out at 60 seconds of inactivity otherwise. I think it's better to keep the stream open, even if the camera is broken. The implementation looks awkward, but that might be the state of Rust async right now.
2021-03-26 02:01:38 -04:00
tokio-stream = "0.1.5"
upgrade tokio-tungstenite
2022-03-08 14:39:22 -05:00
tokio-tungstenite = "0.17.1"
tweak config format (#133) * switch from json to toml. I think this will be more user-friendly. It allows comments and has less punctuation. Fewer surprises than yaml (which has e.g. the "Norway problem"). I might have stayed with JSON if I could see a good serde json library that allows comments, but hson is unmaintained and serde-json strictly follows the spec. * switch from camelCase to snake_case. Seems more idiomatic for TOML and matches the Rust source. * forbid unknown keys. Better to spot errors sooner. * rename "trust_forward_hdrs" to "trust_forward_headers". Nothing else is abbreviated.
2022-03-16 15:28:08 -04:00
toml = "0.5"
logging improvements * allow debug/trace logging on release builds again * enable log messages from hyper. I didn't notice they went away with 0.14.0, although there's a breaking change in the log: https://github.com/hyperium/hyper/blob/master/CHANGELOG.md#v0140-2020-12-23 * downgrade some particularly spammy messages
2021-06-05 02:28:48 -04:00
tracing = { version = "0.1", features = ["log"] }
upgrade some deps, including reqwest The reqwest one is particularly notable because it means not having two versions of hyper/http/tokio/futures/bytes. It also drops a number of transitive deps; with some work I think I could stop depending on regex now.
2020-01-09 23:06:30 -05:00
url = "2.1.1"
uuid = { version = "0.8", features = ["serde", "std", "v4"] }
fully implement json handling as in spec This is a significant milestone; now the Rust branch matches the C++ branch's features. In the process, I switched from using serde_derive (which requires nightly Rust) to serde_codegen (which does not). It was easier than I thought it'd be. I'm getting close to no longer requiring nightly Rust.
2016-12-09 00:28:50 -05:00
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
[dev-dependencies]
drop ffmpeg support * switch the config interface over to use Retina and make the test button honor rtsp_transport = udp. * adjust the threading model of the Retina streaming code. Before, it spawned a background future that read from the runtime and wrote to a channel. Other calls read from this channel. After, it does work directly from within the block_on calls (no channels). The immediate motivation was that the config interface didn't have another runtime handy. And passing in a current thread runtime deadlocked. I later learned this is a difference between Runtime::block_on and Handle::block_on. The former will drive IO and timers; the latter will not. But this is also more efficient to avoid so many thread hand-offs. Both the context switches and the extra spinning that tokio appears to do as mentioned here: https://github.com/scottlamb/retina/issues/5#issuecomment-871971550 This may not be the final word on the threading model. Eventually I may not have per-stream writing threads at all. But I think it will be easier to look at this after getting rid of the separate `moonfire-nvr config` subcommand in favor of a web interface. * in tests, read `.mp4` files via the `mp4` crate rather than ffmpeg. The annoying part is that this doesn't parse edit lists; oh well. * simplify the `Opener` interface. Formerly, it'd take either a RTSP URL or a path to a `.mp4` file, and they'd share some code because they both sometimes used ffmpeg. Now, they're totally different libraries (`retina` vs `mp4`). Pull the latter out to a `testutil` module with a different interface that exposes more of the `mp4` stuff. Now `Opener` is just for RTSP. * simplify the h264 module. It had a lot of logic to deal with Annex B. Retina doesn't use this encoding. Fixes #36 Fixes #126
2022-03-18 13:30:23 -04:00
mp4 = { git = "https://github.com/scottlamb/mp4-rust", branch = "moonfire" }
upgrade some server deps I avoided rtcp 0.2.2->0.2.3 because of an accidental semver break.
2021-07-09 18:01:15 -04:00
num-rational = { version = "0.4.0", default-features = false, features = ["std"] }
upgrade tokio, bytes, hyper, reqwest, http-serve This was mostly straightforward. The most confusing part waas the Sync bound change on body streams. I copied what hyper did and it seemed to work. /shruggie
2021-01-27 14:47:52 -05:00
reqwest = { version = "0.11.0", default-features = false, features = ["json"] }
stop using old tempdir crate
2021-05-17 16:08:01 -04:00
tempfile = "3.2.0"
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
always optimize scrypt password hashes
2020-11-24 10:53:52 -05:00
[profile.dev.package.scrypt]
# On an Intel i3-6100U @ 2.30 GHz, a single scrypt password hash takes 7.6
# seconds at opt-level=0, or 0.096 seconds at opt-level=2. Always optimize this
# crate to avoid seeming hung / being annoyingly slow when debugging.
opt-level = 2
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
[profile.release]
reduce debug output in release/bench builds This reduces the binary size from 154 MiB to 70 MiB (#70 progress). Tools like `cargo flamegraph` still work fine. As suggested by "EarthFeet" on reddit: https://www.reddit.com/r/rust/comments/rw0jad/cargos_strip_profile_option_has_been_stabilized/hra193k/
2022-01-05 12:55:30 -05:00
debug = 1
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
build docker image with lto
2022-04-14 13:53:03 -04:00
[profile.release-lto]
inherits = "release"
lto = true
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
[profile.bench]
reduce debug output in release/bench builds This reduces the binary size from 154 MiB to 70 MiB (#70 progress). Tools like `cargo flamegraph` still work fine. As suggested by "EarthFeet" on reddit: https://www.reddit.com/r/rust/comments/rw0jad/cargos_strip_profile_option_has_been_stabilized/hra193k/
2022-01-05 12:55:30 -05:00
debug = 1