fix(audio): offset the analysis lookup by the track's placement - #198
Merged
Conversation
An audio track with start: 73 played the file from its beginning at t=73 in the video, while a waveform or audio_spectrum at t=73 drew the file's content *at 73 seconds*. Picture and sound disagreed by exactly `start`. Two independent time bases: encode/audio.rs places the track at track.start * TARGET_SAMPLE_RATE and copies the file from its own sample 0, while the analysis indexes the file from its own frame 0 and the painters call amplitude_at(ctx.time) with scenario time. Carry `start`/`end` in AudioAnalysis and apply them in one place, `track_time`, which every accessor now goes through. The consumers are unchanged: the cache is keyed by path and the painters never see the AudioTrack, so the knowledge belongs with the data rather than at three call sites. Outside [start, end) the accessors return 0, so a visualisation goes flat exactly when the track is silent instead of drawing an envelope nobody hears. The cache fingerprint gains the placement: the analysis content depends only on the file, but the lookup no longer does, so an entry computed for one placement must not be reused for another.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #190.
What
An
audiotrack withstart: 73played the file from its beginning at t=73 in the video. Awaveformoraudio_spectrumat t=73 drew the file's content at 73 seconds. Picture and sound disagreed by exactlystart.Two independent time bases:
encode/audio.rsplaces the track attrack.start * TARGET_SAMPLE_RATEon the scenario timeline and copies the file from its own sample 0;encode/audio_analysis.rsanalyses the raw file, indexed from its own frame 0, and the painters callamplitude_at(ctx.time)with scenario time.track.startappeared nowhere in that file.endhad the mirror problem: the analysis kept running past the point where the track was cut.Why it matters in practice
Gating a soundtrack to one scene is a natural thing to want, and the obvious way to write it —
start/end— silently desynchronised the visualisation. The workaround isvolume_keyframes, which leaves the time mapping intact but is not what the fields suggest.How
AudioAnalysiscarriesstart/end, and every accessor goes through onetrack_time(scenario_time) -> Option<f64>:Noneoutside the window (so the visualisation goes flat exactly when the track is silent),Some(time - start)inside.The three consumers —
waveform,audio_spectrum, and theaudio-reactivebinding inbox_builder— are unchanged. The cache is keyed by path and the painters never see theAudioTrack, so the placement belongs with the data rather than duplicated at three call sites.The cache fingerprint gains the placement. The analysis content depends only on the file, but the lookup no longer does, so an entry computed for one placement must not be reused for another — otherwise opening a second scenario that reuses the same file at a different offset would silently keep the first one's.
Not in scope
volume_keyframesand the fades are a separate question — the analysis is of the source, not of the mix — and now more visible since #182 made the studio play the mixed output. Worth deciding explicitly; not decided here.Test
a_track_start_offsets_the_analysis_lookup: 1 s of sine then 1 s of silence, placed atstart: 5.0,end: 6.5. Asserts flat before 5, loud at 5.2 (0.2 s into the file), quiet at 6.2 (1.2 s in, the silence), flat past 6.5 — and that the smoothed and band accessors take the same path, or a bound component would disagree with the waveform beside it.cargo test --workspacegreen,cargo fmt --checkandcargo clippy --all-targets -- -D warningsclean.