Commit Graph

28 Commits

Author SHA1 Message Date
Scott Lamb
f01f523c2c refine 1->3 upgrade process
In hindsight, the "post_tx" step in the upgrade process introduced in
e7f5733 doesn't make sense. If the procedure fails at this stage, nothing says
it still needs to be completed. If the sample file dirs have to be updated
after the database, then there should be another database version to mark that
it's fully completed, and indeed that's the purpose version 3 serves. So get
rid of the Upgrader trait and just go back to a simple run function per
version.

In the case of the sample file dir metadata, it actually can happen before the
database transaction; the stuff written to the database later just needs to be
consistent with what it finds if there's an existing metadata file from a
half-completed update.

For safety, ensure there are no unexpected directory contents before
upgrading 1->2, and ensure the metadata matches before upgrading 2->3.
2018-03-01 09:47:56 -08:00
Scott Lamb
bcf42fe02c move db upgrade logic into db crate
This allows shrinking db's API surface.
2018-02-28 21:21:47 -08:00
Scott Lamb
fbe1231af0 move open_id from recording_playback to recording
I want to be able to use it in etags without having to do a full scan of the
recording_playback in advance, which would greatly increase time to first
byte. I probably will even use it in urls to ensure the segments they point to
are stable. I haven't actually done this yet - it will wait until I implement
serving unflushed recordings - but I want to get the schema set up properly.
2018-02-28 20:52:43 -08:00
Scott Lamb
bf45ae6011 extend recording_playback with an open_id
As noted in schema.sql, this can be used for disambiguation. It also may be
useful in diagnosing data integrity problems.

Also, sneak in a couple minor improvements: better diagnostics in a couple
places, fix to 1->2 upgrade procedure.
2018-02-22 21:46:41 -08:00
Scott Lamb
b037c9bdd7 knob to reduce db commits (SSD write cycles)
This improves the practicality of having many streams (including the doubling
of streams by having main + sub streams for each camera). With these tuned
properly, extra streams don't cause any extra write cycles in normal or error
cases. Consider the worst case in which each RTSP session immediately sends a
single frame and then fails. Moonfire retries every second, so this would
formerly cause one commit per second per stream. (flush_if_sec=0 preserves
this behavior.) Now the commits can be arbitrarily infrequent by setting
higher values of flush_if_sec.

WARNING: this isn't production-ready! I hacked up dir.rs to make tests pass
and "moonfire-nvr run" work in the best-case scenario, but it doesn't handle
errors gracefully. I've been debating what to do when writing a recording
fails. I considered "abandoning" the recording then either reusing or skipping
its id. (in the latter case, marking the file as garbage if it can't be
unlinked immediately). I think now there's no point in abandoning a recording.
If I can't write to that file, there's no reason to believe another will work
better. It's better to retry that recording forever, and perhaps put the whole
directory into an error state that stops recording until those writes go
through. I'm planning to redesign dir.rs to make this happen.
2018-02-22 16:35:34 -08:00
Scott Lamb
31adbc1e9f initial split of database to a separate crate
It should reduce compile time / memory usage to put quite a bit of the code
into a separate crate. I also intend to limit visibility of some things to
only within the db crate, but that's for a future change. This is the smallest
move that will compile.
2018-02-20 23:15:39 -08:00
Scott Lamb
d84e754b2a replace homegrown Error with failure crate
This reduces boilerplate, making it a bit easier for me to split the db stuff
out into its own crate.
2018-02-20 22:46:14 -08:00
Scott Lamb
253f3de399 reorganize the sample file directory
The filenames now represent composite ids (stream id + recording id) rather
than a separate uuid system with its own reservation for a few benefits:

  * This provides more information when there are inconsistencies.

  * This avoids the need for managing the reservations during recording. I
    expect this to simplify delaying flushing of newly written sample files.
    Now the directory has to be scanned at startup for files that never got
    written to the database, but that's acceptably fast even with millions of
    files.

  * Less information to keep in memory and in the recording_playback table.

I'd considered using one directory per stream, which might help if the
filesystem has trouble coping with huge directories. But that would mean each
dir has to be fsync()ed separately (more latency and/or more multithreading).
So I'll stick with this until I see concrete evidence of a problem that would
solve.

Test coverage of the error conditions is poor. I plan to do some restructuring
of the db/dir code, hopefully making steps toward testability along the way.
2018-02-20 10:11:10 -08:00
Scott Lamb
e7f5733f29 new database/sample file dir interlock scheme
The idea is to avoid the problems described in src/schema.proto; those
possibilities have bothered me for a while. A bonus is that (in a future
commit) it can replace the sample file uuid scheme in favor of using
<camera_uuid>-<stream_type>/<recording_id> for several advantages:

  * on data integrity problems (specifically, extra sample files), more
    information to use to understand what happened.
  * no more reserving sample files prior to using them. This avoids some extra
    database transactions on startup (now there's an extra two total rather
    than an extra one per stream). It also simplifies an upcoming change I
    want to make in which some streams are not flushed immediately, reducing
    the write load significantly (maybe one per minute total rather than one
    per stream per minute).
  * get rid of eight bytes per playback cache entry in RAM (and nine bytes
    per recording_playback row on flash).

The implementation is still pretty rough in places:

  * Lack of tests.
  * Poor ode organization. In particular, SampleFileDirectory::write_meta
    shouldn't be exposed beyond db. I'm thinking about moving db.rs and
    SampleFileDirectory to a new crate, moonfire_nvr_db. This would improve
    compile times as well.
  * No tooling for renaming a sample file directory.
  * Config subcommand still panics in conditions that can be reasonably
    expected to happen.
2018-02-14 23:35:52 -08:00
Scott Lamb
89b6bccaa3 support multiple sample file directories
This is still pretty basic support. There's no config UI support for
renaming/moving the sample file directories after they are created, and no
error checking that the files are still in the expected place. I can imagine
sysadmins getting into trouble trying to change things. I hope to address at
least some of that in a follow-up change to introduce a versioning/locking
scheme that ensures databases and sample file dirs match in some way.

A bonus change that kinda got pulled along for the ride: a dialog pops up in
the config UI while a stream is being tested. The experience was pretty bad
before; there was no indication the button worked at all until it was done,
sometimes many seconds later.
2018-02-11 23:04:02 -08:00
Scott Lamb
6f309e432f store rfc6381_codec in the database
This avoids having codec-specific logic to synthesize it in db.rs. It's not
too much of a problem now with only H.264 support, but it'd be a pain when
supporting H.265 and other codecs.
2018-02-05 11:57:59 -08:00
Scott Lamb
cc6579b211 upgrader for v1->v2 2018-02-03 22:19:02 -08:00
Scott Lamb
57c44b5e35 schema 2: add a "record" bool to streams 2018-02-03 22:19:02 -08:00
Scott Lamb
dc402bdc01 schema version 2: support sub streams
This allows each camera to have a main and a sub stream. Previously there was
a field in the schema for the sub stream's url, but it didn't do anything. Now
you can configure individual retention for main and sub streams. They show up
grouped in the UI.

No support for upgrading from schema version 1 yet.
2018-02-03 22:15:54 -08:00
Scott Lamb
5c8970fe8a update dependencies 2017-11-16 23:01:09 -08:00
Scott Lamb
8de7e391f8 populate timeZoneName as expected by UI
This works by a nasty hack, but it seems to work well enough for now.
Fingers crossed.
2017-10-21 23:57:13 -07:00
Scott Lamb
315f3594c2 add a basic Javascript UI
The Javascript is pretty amateurish I'm sure but at least it's something to
iterate from. It's already much more pleasant for browsing through videos in
several ways:

* more responsive to load only a day at a time rather than 90+ days
* much easier to see the same time segment on several cameras
* more pleasant to have the videos load as a popup rather than a link
  that blows away your position in an enormous list
* exposes the fancier .mp4 generation options: splitting at lengths
  other than the default, trimming to an arbitrary start and end time,
  including a subtitle track with timestamps.

There's a slight regression in functionality: I didn't match the former
top-level page which showed how much camera used of its disk allocation and
the total duration of video. This is exposed in the JSON API, so it shouldn't
be too hard to add back.
2017-10-21 21:54:27 -07:00
Scott Lamb
11420df065 update deps (particularly hyper) + fix warnings 2017-09-21 21:51:58 -07:00
Scott Lamb
8ff1d0dcb8 workaround config crash since cursive 0.5.1
https://github.com/gyscos/Cursive/issues/144
2017-07-02 22:03:16 -07:00
Scott Lamb
bebd6ee79a update dependencies
* The mylog update fixes a couple bad bugs.
* Otherwise, just keep up with the Rust ecosystem.
2017-06-11 12:57:55 -07:00
Scott Lamb
bfc0e2abe8 use my own logging package
This supports formats that I find more useful; one that mimicks the Google
glog package, and one that is similar but adapted for the systemd journal.
2017-03-26 00:01:48 -07:00
Scott Lamb
1cf27c189f upgrade to async hyper
serve_generated_bytes is >3X faster. One caveat is that the reactor thread may
stall when reading from the memory-mapped slice. Moonfire NVR is basically a
single-user program, so that may not be so bad, but we'll see.
2017-03-02 19:29:28 -08:00
Scott Lamb
ce363162f4 trim 16 bytes from each recording::Segment
This reduces the working set by another 960 bytes for a typical one-hour recording, improving cache efficiency a bit more.

8 bytes from SampleIndexIterator:
   * reduce the three "bytes" fields to two. Doing so as "bytes_key" vs
     "bytes_nonkey" slowed it down a bit, perhaps because the "bytes" is
     needed right away and requires a branch. But "bytes" vs "bytes_other"
     seems fine. Looks like it can do this with cmovs in parallel with other
     stuff.
   * stuff "is_key" into the "i" field.

8 bytes from recording::Segment itself:
   * make "frames" and "key_frame" u16s
   * stuff "trailing_zero" into "video_sample_entry_id"
2017-02-27 21:14:06 -08:00
Scott Lamb
b7957edb5a fix "Edit retention" fs capacity calculation
This was completely wrong: it overflowed on large filesystems and
double-counted the used bytes.

The new logic is still imperfect in that if there are a bunch of files in the
process of being deleted (moved from recording to reserved_sample_files but
not yet unlinked), they'll be taken out of the total capacity. Maybe it should
stat everything in the sample file directory instead of relying on the
recording table. It's definitely an improvement, though.
2017-02-12 20:45:46 -08:00
Scott Lamb
da4e439b9c benchmark camera page, fix broken schema
This page was noticeably slower than necessary because the recording_cover
index wasn't actually covering the query. Both the schema for new databases
and the upgrade query were broken (and not even in the same way).

No new schema version to correct this, at least for now. I'll probably have
another reason to change the schema soon anyway and can throw this in.
2017-02-12 20:37:03 -08:00
Scott Lamb
c82f038bef new "moonfire-nvr config" subcommand
This is a ncurses-based user interface for configuration. This fills a major
usability gap: the system can be configured without manual SQL commands.
2017-02-05 19:58:41 -08:00
Scott Lamb
168cd743f4 new command to initialize a database 2017-01-17 14:21:13 -08:00
Scott Lamb
3af9aeee96 use xsv-style subcommands like "moonfire-nvr run"
This makes it easier to understand which options are valid with each
command.

Additionally, there's more separation of implementations. The most
obvious consequence is that "moonfire-nvr ts ..." no longer uselessly
locks/opens a database.
2017-01-17 12:51:56 -08:00