Ask about a song
Answers about the songs come from the notes in this project. When the notes do not cover something, the answer says so.
Try one of these
The system
A singer sings. A browser, a Python server and a small family of local models listen, decide, and answer in a synthetic voice. This page is the map: every process, every stage, the contract between them, and the reason each one is built the way it is.
01 The shape of the system
The project is one codebase with three front doors. The live path listens to a microphone and answers in real time. The SingSong path listens to a finished vocal and composes a band around it. The MySong path takes a melody, reads its harmony, and arranges a band from chords. All three share the same pitch, rhythm, key and audio code, which is why a fix in one makes the others better.
02 The live path
The live path is a chain of small, replaceable stages. Each stage has one job and one output shape, and the next stage never reaches back into the previous one. That is what makes it possible to swap YIN for RMVPE, or a DSP voice for a neural one, without rewriting the performance.
| Stage | File | Input → output | What it guarantees |
|---|---|---|---|
| Capture | core/audio_capture.py | device → 1024-sample float32 frames | The callback only enqueues; a slow pipeline can never starve the audio thread. |
| Gate | core/preprocessor.py | frame → voiced frame or silence | Room noise below −40 dBFS never becomes a note; level is normalised before analysis. |
| Pitch | analysis/pitch_detector.py | frame → Hz + confidence | 65–1200 Hz, two hops per frame, jump-aware smoothing; YIN or RMVPE behind the same interface. |
| Rhythm | analysis/rhythm_analyzer.py | 2 s buffer → onsets, BPM, phrase state | Onsets are computed once per second of new audio, not once per frame, so a note is counted once. |
| Phonetics | analysis/phonetic_analysis.py | voiced frame → brightness, colour, nasality | 13 MFCCs + centroid; a trained classifier if present, documented heuristics if not. |
| Key | synthesis/harmony_engine.py | 256-pitch history → root + mode | Krumhansl–Kessler correlation, re-estimated every 43 frames (about once a second), never per note. |
| Decision | synthesis/accompaniment_modes.py | context → mode, target Hz, action | Protocol silence is checked before anything musical; the rest is an ordered checklist. |
| Voice | synthesis/vocable_synthesizer.py | decision → stereo audio | Vowel, texture and vibrato are applied here; the neural stage receives separate stems, never a mix. |
| Timing | core/timing_sync.py | note → scheduled playback | 70 ms base delay, beat snapping, stale notes dropped rather than played late. |
The browser demo uses the same analysis maths but not the hardware TimingSync; it schedules Web Audio directly. Both are real, they are simply different transports for the same decision.
03 Frame budget
At 44,100 Hz a 1024-sample frame is 23.2 ms of audio. Every real-time stage has to finish well inside that window, or the next frame is already waiting. The design keeps the expensive parts off the critical path instead of hoping they get faster.
YIN on two 512-sample hops, one MFCC pass, a few deque updates, a key lookup, and a mode decision. No disk, no network, no model loading, no allocation of a new graph.
RMVPE runs on a rolling window every few hops. Neural voice conversion lives in its own process. SingSong and MySong render an uploaded file after the fact, where a two-second wait is acceptable because nobody is singing into it.
04 Runtime
The app is not a monolith at runtime, even though it is one repository. Each boundary below exists because something inside it is slow, optional, or likely to fail in a way that must not take the performance down with it.
05 The stack
A stack table is only worth reading if it says why. Each row below names the alternative that was not chosen and the consequence of choosing this one.
archer:14b, and it answers conversation, song-note questions, translation and script conversion; it is installed from a single export. Ollama manages that model's lifecycle; llama.cpp is the single-process alternative for the same GGUF.06 Decisions
Architecture is mostly a record of what you refused to do. These are the decisions that still constrain the code today, written as decision, reason and consequence rather than as a feature list.
Every model runs on the same computer as the microphone. The consequence is real: no API keys, no per-note cost, no network jitter in the audio path. It also means the project ships code, not a hosted service, and the person running it owns the data.
RVC and the language model are sidecars. If the voice model takes 2.5 seconds to convert a note, the DSP voice still answers on time; if Ollama is not running, the chat reports it and the stage keeps singing. The boundary is a timeout, not a promise.
YIN is the default because it fits the frame budget on a CPU. RMVPE is available when accuracy matters more than latency, with a rolling window and a cached result between hops. One interface, two behaviours, chosen live.
Neural conversion is expensive; a note is not. The bank is rendered once, offline, then pitch-shifted and cached by semitone at runtime. The singer hears a neural timbre without waiting for a neural network on every note.
A choir mixed down to one signal gives the voice model several simultaneous pitches and a smeared result. Converting each detuned stem separately keeps every call monophonic, and lets harmony stems rotate across voices instead of cloning one.
Waveform generation at 16 kHz for ten seconds is a very long sequence. The paper's factorization into semantic, coarse and fine tokens makes the problem tractable, and each stage can be trained and evaluated on its own.
A chord sequence is a path through a small state space, so exact inference is possible and reproducible. The semi-Markov decoder can change chord on any beat, which fixed one-chord-per-bar grids could not express.
07 Live demonstration
This is the real pitch stage, not a recording. YIN runs in the browser through Web Audio and never sends audio anywhere; RMVPE runs on the server and streams the microphone over a WebSocket. The interval buttons show what the harmony stage would do with the note it hears.
The demo reports the nearest equal-tempered note and the cents deviation. A reading inside ±15 cents is drawn as in tune; the bar shows the raw deviation before any smoothing.
08 Where to go next
The Harmony page opens the decision engine and its ten modes. Vocable Synthesis follows a decision into actual samples, formants and the neural voice. Cree Sentiment covers the language layer, Archer Bot covers retrieval, and Measurements shows every number on this site being produced by a reproducible test.
Harmony · Vocable Synthesis · Cree Sentiment · Archer Bot · Measurements
Harmony
A fixed third above the melody is one idea, repeated forever. This engine has ten musical functions instead, and a public rule for choosing between them. The first rule is not musical at all: Archer decides whether the robot may sing.
01 The premise
Traditional harmonisers assume the answer to “what should the second voice sing?” is a pitch a fixed interval away. That works for a hymn. It does not work for a solo voice carrying a phrase, because a second voice that doubles every note at a third is no longer accompanying; it is competing for the same musical role.
The modes here are best read as functions, not intervals. Unison shadowing reinforces identity. Drone support holds a floor. Call and response gives the singer space and then answers into it. Timbral thickening changes colour without changing pitch. Each one makes a different claim on the listener's attention, and the engine's job is to pick the claim that fits the moment.
02 The vocabulary
Five of these are chosen automatically, three are manual or configured defaults, one is fusion-only, and one is the stop. The distinction is not cosmetic: it is the difference between a behaviour the engine can infer from context and a behaviour a person has to ask for.
Sings the singer's exact pitch in another voice. No interval, no scale snap. The identity is reinforcement: the same note, wider. It is the default because it is the least presumptuous thing a second voice can do.
Holds the same pitch but changes the vowel, brightness and attack, so the double becomes a colour rather than a copy. Triggered after a note has been sustained for thirty frames, where a second onset would sound like an error and a pitch change would sound like harmony the singer did not ask for.
While the melody holds still, it sits in unison. When the pitch genuinely moves, it glides a step above a rising line or below a falling one, then settles back when the motion stops. Hysteresis keeps it from flickering at the threshold; a continuous timbre lean tracks the shape even when the pitch does not move.
One note, one vowel, one breath per phrase, locked near the tonic and held under the line. It is not “the melody an octave down”; it waits for a phrase to begin, opens once, and stops when the phrase stops. The pitch is clamped into a singable low register so it stays a voice rather than a foghorn.
When the singer stops, the answer begins: a phrase with the same rhythm as the call, arriving on the tonic by step. The live responder composes it from the last phrase; the offline renderer composes a better one from the whole take, because it can see the ending before it plays it.
Doubles the line an octave down, with the three things a singer actually does on a low note: a brighter vowel, a narrower vowel, and less level. Without those, an octave double reads as the same voice gone muddy rather than a second singer joining underneath.
Sings the pitch the singer used one beat ago. The decision supplies the note; a separate echo effect supplies the repeating, decaying taps. Keeping those apart is deliberate: a delay pedal repeats whatever came out of it, and does not need to know the note.
One closed-mouth voice, barely there. It is the quietest possible presence: not a harmony, not a double, just another body in the room. Useful when the arrangement needs company without a musical statement.
A conventional chord tone above the melody, unlocked only when fusion mode is explicitly on. It is the one mode that behaves like a traditional harmoniser, and it is opt-in because that vocabulary belongs to a different musical context.
The stop. It returns no pitch, no note and no sound, and it is checked before every other branch. If sovereignty is off, or the current phoneme belongs to the protocol-sensitive set, the answer is silence even if a mode button is lit.
03 Selection
There is no scoring function and no learned policy here. Selection is a short ordered list of questions; the first one that answers wins. That makes the behaviour inspectable: given the same context, you can predict the mode by reading six lines.
The same six checks, in the same order, as AccompanimentModeSelector.select. Toggle a condition and watch a later branch become unreachable.
04 Key and scale
Modes that invent a pitch (contour following, the fusion chord) are rounded to the nearest degree of the inferred key before anything is sung. Modes that return the singer's own note are exempt: rounding a unison is how a double ends up a few cents beside the singer instead of with them.
The key is inferred once a second from the last 256 pitched frames. The distribution of pitch classes is correlated against Krumhansl–Kessler major and minor profiles, and the best-fitting root and mode become the current key. The drone is anchored to that root, not to whatever note just passed; the call-and-response answer is built from its scale degrees, not rounded into them afterwards.
Scale lock is therefore a small, explicit exception list, not a blanket rule. It touches contour following and triadic harmony. It does not touch unison, octave, delay, timbral thickening or drone. The reason is simple: those five already have a trustworthy pitch, and the two that do not are the two that calculate.
05 Voice leading
When a mode needs more than one note, the voicing search prefers the inversion that moves each voice the shortest distance from where it already was. That single constraint is what makes a sequence of chords sound like voices holding a conversation instead of a keyboard being pressed.
The test suite generates 120 progressions, voices each chord with the project's search and with a naive root-position baseline, and measures the total semitone movement of the upper voices plus the number of parallel perfect fifths and octaves. The search cuts mean movement from 1.56 to 0.90 semitones per transition and parallel perfects from 3.89 to 3.61 per hundred transitions, while keeping a median span of nine semitones. The improvement is measured on the same progressions the baseline sees.
06 Sovereignty
The protocol guard is checked before the key, before the tempo, before the phrase, and before any mode's own logic. When it is off, the engine returns silence with a reason, not a quiet note. When the current phoneme is in the protected class list, the same thing happens even with a mode button lit.
This is a political requirement expressed as control flow. The robot is a guest in Archer's performance. A guest who can only be asked to leave after the harmony has already been chosen is not a guest; it is a collaborator with veto power over silence. Putting the check first makes silence the default state and sound the exception that has to be earned.
07 Live demonstration
Pick a mode, allow or deny sovereignty, and sing. YIN runs in the browser; RMVPE can run on the server. The readout shows the note the robot would answer with under that function, not a pre-recorded example.
Vocable synthesis
The robot does not speak Cree. It sings vocables: open vowels shaped like the ones Archer uses, pitched to the note the harmony engine chose, and optionally re-skinned through a synthetic voice model. This page follows that sound from a bank of recordings to the speaker.
01 The material
A vocable is a sung syllable that carries no dictionary meaning: aah, ohh, mmm. It is the oldest thing a voice can do with a melody, and it is the right unit here for two reasons.
It stays out of the language's way. Archer's practice is grounded in Cree singing, and the project includes real Cree translation and sentiment tools elsewhere. The robot's voice is not one of them. Making it sing recognizable Cree words would put a synthetic speaker inside a language conversation it has no standing in. A vocable can echo the resonance of Cree vowels without claiming to pronounce them.
It is the smallest unit that can be shaped musically. A vowel has a pitch, a loudness, an attack and a spectral envelope, and every one of those can be controlled independently. A word brings consonants, timing and meaning, and every one of those would have to be modelled before a single note could be sung. The vocable keeps the problem at the size of the voice.
02 Three engines
The harmony engine emits a note, a vowel, a texture and an action. What turns that into samples is a separate choice, set once in config.yaml. The three engines trade realism against latency, and the default is the one that keeps a live performance live.
| Engine | What it plays | Cost per note | Why choose it |
|---|---|---|---|
neural_wavetabledefault | A short take already converted through the trained voice once, offline, then pitch-shifted to the target note. | One cached lookup and a resample. | Neural timbre at live speed. No network call on the hot path. The cache is keyed by the nearest semitone, so a repeated note is free. |
wavetable | A single recorded take per vowel, pitch-shifted live. | One resample. | Simple and predictable, with no neural dependency at all. Useful for testing the rest of the chain without a model. |
sinusoidal | A source built from four stacked formant resonances, synthesised from scratch. | Four filters per note. | No files to lose, no model to install, and every parameter is inspectable. It is the reference implementation of the vowel. |
03 The bank
A live neural conversion takes about two and a half seconds on this machine. A note lasts a fraction of that. Running the model per note would make the robot late for every phrase, so the bank inverts the problem: convert a small set of takes ahead of time, then shift them at runtime.
04 The vowel
A vowel is not a frequency. It is a pattern of resonances: peaks in the spectrum that the throat and mouth impose on whatever the vocal folds are doing. The first two peaks, F1 and F2, do most of the work. Move them and aah becomes ee without changing the pitch at all.
The synthesizer treats the vowel as a small set of targets, not a recording. F1 and F2 are blended toward the active phoneme's values, a high shelf adds brightness, and a notch-and-peak pair shapes nasality. The singer's own MFCCs and spectral centroid drive those targets in heuristic mode, with an influence of 0.6, so the robot's mouth follows the singer's mouth rather than sitting at a neutral midpoint.
This is a browser model of the same source–filter idea: a sawtooth source through three bandpass formants, with a vibrato LFO. It is simple enough to hear each control separately. The Python engine adds the bank, the phoneme tracking and the ensemble.
05 The ensemble
A choir preset is not a louder solo. It is several independently rendered layers, each with its own small detune, onset jitter and vocal-tract scale, mixed together and sent through a short algorithmic reverb. The differences are what the ear reads as separate people.
The texture is chosen per mode. Unison defaults to choir, octave to duet, delayed response and call-and-response to solo, contour and drone to duet. The parameters live in synthesis.texture: voice count, detune spread, formant spread, jitter, reverb and level. A thirty-millisecond crossfade removes clicks at note boundaries, and the reverb is built from a handful of comb and allpass filters rather than an impulse response, so it costs almost nothing and never needs a file.
The detune is not random per note. It is locked per phrase, so a voice keeps its own slightly sharp or flat identity instead of wandering between notes. That is the difference between a section and a chorus effect.
06 The small physics
A synthesised note is not one oscillator. It is an envelope, a slow pitch movement, a little breath, a filter that follows the mouth, and a join to the note before it. Each of those is a small parameter, and each one is audible when it is wrong.
| Parameter | Value | What it does |
|---|---|---|
| Vibrato | 5.3 Hz · 14 cents | Fades in after roughly 0.6 s on a held note. A note that starts with full vibrato sounds like a synthetic pad; one that grows into it sounds sung. |
| Breathiness | 0.045 | A small amount of formant-shaped noise mixed under the tone. It is the difference between a pure harmonic stack and a throat. |
| Note crossfade | 30 ms | Blends the start of a note against the tail of the previous one, so a legato line has no click at the join. |
| Reverb | comb + allpass | Algorithmic, not a sampled impulse. A solo gets a little; a choir gets more. It costs no file and no latency. |
| Phoneme influence | 0.6 | How far the singer's own brightness, vowel colour and nasality pull the robot's formants away from neutral. |
The phoneme profile is measured, not guessed. Thirteen MFCCs and a spectral centroid are extracted from each voiced frame, then mapped to three values: brightness from 0 to 1, vowel colour from front to back, and nasality from oral to fully nasal. With a trained classifier the mapping is learned; without one, the heuristic rules in analysis/phonetic_analysis.py do the same job. The synthesizer only ever sees the three numbers, so the same profile can drive a formant synth, a bank or a neural voice without changing the analysis code.
07 The neural voice
RVC is a voice conversion model: it takes audio and a target voice model, and returns the same performance in that model's timbre. Here it runs in a separate process on port 8801, so a two-and-a-half-second conversion can never block the audio loop. The configured models are separate trained voices, not recordings of Archer.
The important detail is that the converter receives stems, not the mix. A solo voice is one stem. A duet is two. A choir is several, each slightly detuned. If the ensemble were mixed first, the model would hear several pitches at once and produce a smeared, unstable conversion. Keeping the stems separate means every call is monophonic, which is the only condition the model was trained for.
The lead always converts through the voice model the performer chose. Harmony stems rotate across the other configured voices, so a neural choir sounds like a room of different people rather than one person cloned at several pitches. The sidecar processes one conversion at a time, so a live take caps how many ensemble voices are converted; an offline render can afford to convert every one.
| Cold | Source takethe recorded voice, before conversion | Neural takethe same performance through the trained voice model |
|---|---|---|
| Lalanana | Source takethe recorded voice, before conversion | Neural takeconverted with the same model |
These are the bank's own source and converted takes, not a staged comparison. The converted files are shorter because the bank trims them to the sung portion.
08 What to listen for
Attack. Does the note begin with a consonant-like click, or with breath? The 30 ms crossfade and the per-voice jitter exist for this. Vowel. Does the double sound like a different mouth, or the same mouth doubled? Formant spread is the control. Blend. Can you hear the lead above the ensemble without the ensemble disappearing? That is the level and reverb balance, and it is the difference between a duet and a wall.
Cree sentiment
Three layers sit between a sung phrase and a robot that seems to understand it. The phrase becomes text, the text becomes English, and the English becomes an emotional reading. Each layer is separate, each one can be checked on its own, and none of them pretends to be the others.
01 The chain
It would be convenient to hand a microphone to one large multimodal model and ask for “the emotion of this Cree sentence.” It would also make every mistake invisible. Splitting the chain means a transcription error, a translation error and a sentiment error each have a different address.
02 Transcription
Whisper's language="cr" is not a Plains Cree decoder. It selects the English decoder and attaches a Cree language tag, and what comes back for sung Cree is English words that sound vaguely like the vowels. The project measured that, stopped using it for Cree, and built a separate path.
The Cree path uses Meta's MMS-1b-all with both crk adapters, a lexicon and a character model built from this project's own Cree texts, and a CTC beam search constrained so it cannot emit an English word. Before recognition, Demucs can isolate the vocal from the accompaniment, and a harmonic vocal hint keeps the sung pitch content from being discarded by the separator.
English speech still goes through faster-whisper, because for English it is excellent. The demo below lets you choose which language is being spoken. The transcript is shown as it arrives, followed by the checked version, so a rough guess never gets mistaken for a final reading.
Silence is handled explicitly. A span with no voiced frames is not sent to the recogniser at all; it is labelled as silence, which is both faster and clearer than asking a language model to transcribe a breath.
03 Orthography
Plains Cree is written in Roman orthography (SRO), in Canadian Aboriginal Syllabics, and in the anglicised respelling used by learners and singers who did not grow up with either. The app has to move between all three without losing a long vowel or a w.
SRO uses digraphs and diacritics: ê for a long e, â, î, ô, and combinations such as tw or kw that are one sound, not two. Syllabics writes a consonant and its vowel as a single glyph, with a separate final form for a consonant that closes a syllable. A naive character-by-character mapping fails on both.
The converter works from a syllabary table, not from a transliteration service. It normalises macrons to circumflexes, folds optional doubled vowels, syllabifies before mapping, and round-trips every word in the project's corpus back to the exact SRO it started from. That round trip is the test: 12,372 words, zero failures, zero syllable-count failures.
Syllables will appear here.
04 Translation
There is no off-the-shelf Plains Cree–English model to fine-tune. The translation layer is a transformer encoder–decoder trained from scratch on the material that could be assembled and permissioned: community-contributed text from the Cree corpus, aligned sentences, and vocabulary from Watkins' 1865 dictionary.
Cree is polysynthetic, so a single word can carry what English needs a clause to say. Tokenisation is therefore the first engineering problem, not an afterthought. The project uses a Cree-aware segmentation step (Morfessor and a finite-state analyser where available) before a shared SentencePiece vocabulary, so the model sees morphemes rather than an arbitrary split of a long word.
At inference, the model decodes with beam search, keeping several candidate sentences alive at each step and returning the highest-scoring complete one. The model is small, because a large model on a small corpus memorises the corpus rather than learning to generalise.
05 Sentiment
Valence says how positive or negative a line is. Arousal says how calm or intense it is. A quiet lament and a furious one are both negative, and they should not produce the same robot. The app reads both axes, from different evidence, and combines them into one of three responses.
The primary model is a pretrained RoBERTa sentiment classifier, mapped from its classes to a −1 to +1 score. VADER's compound score is the second backend, and the response always reports which one ran. Valence is never inferred from pitch: loudness and register cannot tell joy from grief without a harmonic reference, so the app does not pretend they can.
Text arousal combines the model's emotional charge with intensifiers, exclamation marks, capitals and sentence length. Audio arousal combines pitch range, mean loudness and dynamic variance measured from the take. When a recording is present the two are averaged 50/50, so the same sentence whispered and shouted moves.
The three states are asymmetric. Joy takes a major third if arousal is above 0.5, a perfect fifth if it is calm, and a triangle wave with no vibrato. Grief takes a minor third, a plain sine and a 3.5 Hz vibrato, the only state that gets one. Neutral holds a perfect fifth, a sine, no vibrato, and the resting LED. The LED eyes follow the same state: green and wide, indigo and narrow, or muted.
Major third or perfect fifth. Triangle. No vibrato. Green eyes, pupil at its widest.
Minor third. Sine. 3.5 Hz vibrato. Indigo eyes, pupil at its narrowest.
Perfect fifth. Sine. No vibrato. Muted, resting eyes.
06 Live demonstration
The first box is the real translation model. Type a Plains Cree word or phrase and it returns what the model currently produces, not a canned gloss. The second is the full pipeline: microphone, transcript, translation, valence, arousal and the LED state the robot would show.
The mic pipeline accepts English or Cree. Choosing “Speaking Cree” routes the audio through the MMS recogniser; choosing English routes it through faster-whisper. The output panel reports which engine produced the valence.
07 Where the reading goes
The valence and arousal pair drives the interval, oscillator, vibrato and LED state you see above. The same numbers are returned by /api/analyze and /api/sentiment for any other client. The language layer and the live harmony engine are separate on purpose: one reads meaning, the other reads pitch, and keeping them separate means a mistranslation cannot silently change a note.
Archer Bot
It answers questions about Archer's songs from Archer's own notes, and says plainly when the notes do not contain the answer. The same weights translate between English and nêhiyawêwin, write syllabics and explain Cree words, and there is no second chat model behind them. This page covers the retrieval that feeds it, the data it learned from, the three training passes, and the numbers the last run actually measured.
01 Why retrieval
Ask a general language model about a song it has never heard of and it will rarely say “I do not know.” It will produce a plausible answer in a confident tone. For an assistant speaking about a living artist's work, that is not a minor inaccuracy; it is fabrication with a friendly voice.
Retrieval-augmented generation changes the order of operations. The question is first used to search Archer's notes. Only the passages that search actually returns are placed in the prompt, under a header that says what they are. The system prompt instructs the model to answer from those passages, and to say plainly when they do not contain the answer. The model is still capable of ignoring that instruction, which is why the sources come back with every reply: a reader can see what the answer was built from.
02 Indexing
Every .md and .txt file in song_notes/ is read, split into chunks of roughly 900 characters along paragraph boundaries, and embedded into a 384-dimensional vector by all-MiniLM-L6-v2 on the CPU.
That embedding model is a search index, not a chat model. It ranks passages and never writes an answer; the one language model reads whatever the index returns. So the install has one generator, archer:14b, plus this small fixed encoder that ships with the retrieval dependency.
The chunk size is a compromise between two failures. Too large and a passage about the wrong song dilutes the one relevant paragraph; too small and a thought gets cut before it reaches its point. Paragraph-aware splitting keeps whole thoughts together. A paragraph that is longer than one chunk is windowed instead, and the 150-character overlap covers the seam.
Rebuilding every embedding on every start would be wasteful, so the index is incremental. Each file's content is hashed, and the hash is stored in a manifest alongside the vector offsets. On the next start, an unchanged file reuses its cached vectors; only a changed file is re-chunked and re-embedded. Changing one note does not re-read the corpus.
03 Retrieval
The question is embedded with the same model and compared against every stored chunk by cosine similarity. The four closest chunks are returned, but only those above a minimum score of 0.30. A weak match is not included to fill the quota.
Both sides of the comparison are normalised, so the dot product is the cosine. The search is exact, not approximate: there is no vector database and no index structure beyond a single matrix multiply. At the size of a song-notes folder that is faster than maintaining an approximate index, and it cannot miss a passage because of a hash collision.
The same query returns the source filename for each chunk. Those filenames travel back to the browser with the reply, so the assistant's answer is always accompanied by the notes it came from. This is not a citation bolted on afterwards; it is the retrieval result itself.
This calls /api/chat/retrieve, which runs the same embedding and cosine search as the chat endpoint but stops before the language model. It is the part of the system that can be checked without Ollama running.
04 The prompt
Retrieved passages are placed in the <|song|><|note|> block, each under Notes on <song>:, and the question follows as Question:. A fixed system prompt sits above both, describing the assistant's four jobs and its evidence rule.
The system prompt describes one assistant with four jobs: conversation about the project, answers from the notes, translation, and word or script work. When notes are present they are the only evidence for anything about a song. If they do not contain the answer, the model is told to say the notes do not say. There is no redirect to another model, because there is no other model to redirect to.
The last eight turns of the conversation are sent with the question so follow-ups can refer back. The retrieval step runs fresh every time, including on a follow-up, because the subject of the conversation can change mid-thread. The history is context for tone and reference, never a substitute for retrieved evidence.
05 The model
The backend is a configuration choice, not a rewrite. Ollama runs as a local server and manages models by name; llama.cpp loads a GGUF file inside the Python process. Both keep every token on the machine.
Listens on 127.0.0.1:11434 and serves the one tag, archer:14b, a 14-billion-parameter model quantised to Q4_K_M: about 9 GB on disk, 4,096 tokens of context, and the same weights for every job on this page. Create it once from the exported Modelfile, then the app only sends HTTP to localhost. If the server is not running, or the tag is missing, the endpoint says which, with the command that fixes it.
Loads a .gguf file in-process. One less service to start, at the cost of the app holding the model's memory. Useful on a machine where installing a background server is inconvenient, or where the model file is already managed by hand.
Generation is conservative: temperature 0.3, at most 500 new tokens. A grounded answer should sound like a knowledgeable person speaking briefly, not like a model performing fluency. The cap also keeps a reply from turning into an essay that outruns the four passages it was given.
There is one model behind both backends, one tag and one GGUF. Ollama and llama.cpp are two ways to serve it, not two assistants. Section 08 is what that model does and how one prompt selects the job.
06 The voice
If the request asks for audio, the reply text is passed to a local Piper voice and returned as a base64 WAV alongside the text. The chat bubble can then speak the answer in that configured voice.
Piper is optional by design. If no voice file is configured, the reply still arrives as text with a plain note that spoken replies are not set up. The failure is described in the response rather than swallowed or turned into a 500, because a text answer is still a complete answer.
The spoken voice and the sung voice are separate audio models, not chat models. Piper is text to speech; the vocable synthesizer and RVC are singing models. The assistant above is the only language model in the system, and none of these voices is the artist's voice.
07 The other tab
The floating panel has a second tab that records a hummed or sung idea and returns it through a synthetic voice model. It shares the widget, not the pipeline: the audio goes to /api/chat/sing, which reuses the server's loaded neural voice model.
That endpoint takes the recording and converts it through the same neural timbre stage the synthesis page describes. Nothing is separated first; the whole take goes through the converter. It is a different job from answering a question, and treating it as one keeps both paths simpler. The full explanation of the conversion lives on the Vocable Synthesis page.
Retrieval, prompt, local model, optional speech. Returns the reply text and the source filenames.
A recording in, a converted performance out. Reuses the already-loaded neural voice model rather than starting a second one.
08 One model
There is no router and no second chat model. Conversation, song notes, translation, syllabics and word explanations are one set of weights, selected by the task prefix in the prompt.
The earlier design split the assistant in two: a general model for conversation and a Cree model for language. Every question became a routing decision, the install had two tags, and an English question about a Cree word could land on the model that was worse at it. This version puts every behaviour in one instruction set, so the model learns what a translation looks like, what a notes answer looks like and what ordinary conversation looks like in the same adapter.
Retrieval still runs on every question, but it decides whether notes are in the prompt, not which model answers. A passage above the 0.30 relevance floor is attached under Notes on <song>:; below it, the question takes the no-notes path. A translation request is never padded with song prose.
The request is unwrapped before it reaches the model. "How do you say river in Cree?" becomes the training shape with river after the header, and "Write tânisi in Cree syllabics" sends tânisi. The model learned the sentence to act on, not the question that asked for it, and handing it the whole question makes it translate the question instead.
The response reports the task, the model tag and the notes used. task is the prompt shape, not a backend choice: song_notes, chat, translate or orthography. If the one model is not installed, the chat returns a 503 with the ollama create line. There is nothing else to fall back to, and answering as another model would not be this assistant.
Calls POST /api/chat/prompt, which runs the same retrieval and format_prompt as the chat and stops before the language model. The task it reports is the prompt shape, not a different model.
09 The base
A model trained from nothing on a million words has to learn the language and the task at the same time. This one starts from Qwen2.5-14B-Instruct, which already holds a conversation and follows an instruction, and adds a small set of weights, a Cree vocabulary and the instruction mix for all four jobs. The base is the starting point for training, not a second model that ships: the app loads the merged result as archer:14b and nothing else.
The added weights are LoRA adapters: low-rank matrices attached to the attention and feed-forward projections, trained while the base weights stay exactly as they were. The run described here used rank 32 with alpha 64 on one 24 GB card, with the base loaded in 4-bit. That is 80,740,352 trainable parameters, 1.05% of the model. The base weights never change, and the adapter is small enough to checkpoint, move and resume like an ordinary file.
The vocabulary is the part that matters most for a language the base model has barely seen. The base tokenizer spends 4.72 tokens on the average Cree word, splitting it into pieces that were never meant to be read together. A SentencePiece model trained on the Cree corpus, filtered and then grafted onto the existing vocabulary, spends 3.65 on the same held-out text, 22.6% fewer. The run adds 3,947 pieces and takes the vocabulary from 151,665 to 155,620, and the audit that measures the change is part of the run.
A new token cannot start at random. It starts at the average of the base model's embeddings for the pieces it replaces, so a word that was spelled n · ê · hi · yaw · ê · win already means something on its first day. Random rows have to be learned from scratch, and with a corpus this size they never fully are. The table is then frozen. The input and output embedding matrices together hold about 1.1 billion parameters, and in fp32 with gradients and Adam state that is roughly 11 GB, the difference between fitting a 16 GB card and dying on the first step.
10 The data
Everything the model learns comes from material whose licence is recorded. Each source is written down with its licence, dialect and script before a single sentence is used, and the run carries that ledger into its report.
This run kept 1,121,753 words across 65,861 training sentences, 10,934 aligned Cree and English pairs, a 203-pair document-disjoint test split of which 120 are scored, and 57,579 lexicon forms for the validity check. The smoke profile uses a slice of the same pipeline, which is why its report shows smaller numbers.
The licence policy is part of the loader, not a note added afterwards. NoDerivatives material in the EdTeKLA collection is skipped at the source. NonCommercial ShareAlike material is used only while the released model carries that same licence. Historical books are tagged as historical, so they can teach vocabulary and spelling without being mistaken for how the language is written today.
Showing all 13 collections.
| Source | What it contributes | Licence |
|---|---|---|
| EdTeKLA Cree Corpus | Community texts, stories and lessons in SRO and syllabics. 31 texts used, 37 skipped on licence, and 25 bag-of-words files counted for vocabulary only. | CC BY-NC-SA 4.0 by default, per-text READMEs; NoDerivatives subcollections skipped |
| Plains Cree Figurative Language | Sentence pairs written to show idiom and figurative meaning, not just literal glosses. | CC BY 4.0 |
| Cree Wikipedia | 37 articles survive the language filter. Encyclopedic register and modern vocabulary. | CC BY-SA 4.0 |
| FineWeb-2 Cree slices | 192 documents and about 108,000 words of web text, the small public footprint of the language. | ODC-BY 1.0 |
| FinePDFs Cree slices | 733 PDFs and about 1.19 million words of extracted text, the largest source examined. | ODC-BY 1.0 |
| FineTranslations Cree | 582 aligned Cree and English chunks. The English side is machine translation and is treated as such. | ODC-BY 1.0 |
| GlotCC Cree slices | Web-crawled Cree documents, used for coverage of places and names. | ODC-BY 1.0 |
| Tatoeba Cree | 31 short everyday sentences, the register a learner actually needs. | CC BY 2.0 FR |
| GlotStoryBook Cree | 88 story paragraphs with a parallel English telling, useful for narrative and sentence rhythm. | CC BY-NC-SA, per row |
| Internet Archive books | 12 books used from 653 documents and 456,177 Cree words, after 44 failed the dialect or OCR check. Includes Bloomfield's Plains Cree Texts and Sacred Stories of the Sweet Grass Cree. | public domain, printed before 1935 |
| Plains Cree phrase recordings | 64 phrase transcriptions. The audio is not redistributed and is not used for the language model. | research dataset, transcriptions only |
| GiellaLT lang-crk | 57,579 word forms from a community-maintained morphological lexicon for Plains Cree, the list behind the validity measure and a weighting source for the tokenizer. | AGPL-3.0 source; word list used, not the code |
| This project's own data | 709 aligned pairs, Archer's song notes, and 443 historical Watkins 1865 glosses, used for grounded question and answer examples. | project-owned |
Word counts are the text examined before cleaning, except the totals above, which are what survived. All thirteen are written to ledger.json by the loader. Watkins 1865 travels inside the project entry, and anything placed in uploads/ is read and recorded as unspecified until someone labels it.
The corpus is built on the EdTeKLA Indigenous Languages Corpora at the University of Alberta, the Plains Cree Figurative Language corpus by KonradBRG, Cree Wikipedia, the FineWeb-2, FinePDFs, FineTranslations, GlotCC and GlotStoryBook releases, Tatoeba, the Internet Archive's public-domain Cree books, and the GiellaLT lang-crk lexicon and public texts from ALTLab at the University of Alberta, with the Maskwacîs Education Schools Commission and First Nations University. The orthography tables follow GiellaLT's Latn to Cans and Cans to Latn transducers.
The song notes are Archer Pechawis's own. They are read from song_notes/ to build grounded question and answer examples and are not redistributed as a text corpus.
11 Cleaning
More text is not automatically more language. A corpus assembled from the open web arrives with English sentences, OCR noise, the same lesson in four places and a scan whose syllabics never decoded.
Text is split into sentences and checked one at a time. A sentence has to look like Cree in its function words and its character inventory, and the share of words that are recognisably Cree has to clear a floor. A PDF page that is mostly English with a Cree title is dropped, and a bag-of-words file is allowed to contribute vocabulary counts without ever being read as prose.
Duplicates are removed twice. An exact match is easy; a near-duplicate check catches the same sentence with a different line break, a stray hyphen or one changed ending, which is the form most copied lessons arrive in. The split is made by whole document, not by sentence, so a story cannot appear in training and again in the test set.
After the split there is a second leakage pass. Anything in dev or test that is a near-duplicate of a training sentence is moved into training. A benchmark scored on the sibling of a sentence the model memorised is not a held-out score, and a flattering number from that setup is worse than no number at all.
12 Training
Continued pretraining teaches the language. Instruction tuning teaches the shape of the answers. Preference tuning teaches it to prefer the faithful answer over the fluent one.
Continued pretraining reads the cleaned Cree text as packed 1,024-token windows, with no padding wasted between documents. Ten percent of the blocks are English, drawn from a held-out Wikipedia slice, and that replay is the reason the model still follows an English instruction after a week of Cree. Two passes at a learning rate of 2e-4, over 3,751 packed blocks.
Instruction tuning uses the chat template and scores only the assistant's reply. The question is context, not something to reproduce. The examples cover translation in both directions, SRO and syllabics, word glosses, grounded answers from the song notes, questions the notes cannot answer, and ordinary conversation about the project. That mix is what makes one model work for every request instead of one narrow skill. Two passes at 1.5e-4, lower, because over-training this stage flattens every reply into the same shape.
Preference tuning builds its pairs from the training translations. The gold sentence is the chosen answer; the rejected one has a dropped final clause, a flattened long vowel, an English word pushed in, a changed number, or its word order reversed. Those are the failures a fluent model actually makes. The objective is DPO with a beta of 0.1. The reference is a frozen copy of the instruction-tuned adapter, loaded beside the policy, so preference tuning is pulled back towards what instruction tuning built rather than towards the raw base model.
Each stage checkpoints every ten minutes and resumes from the newest checkpoint that has both its trainer state and an intact adapter header. The stage's root folder is written only after training returns, so an interrupted stage never looks finished. That is what makes a dropped SSH session a pause rather than a restart.
These are the defaults in train_cree_llm.py. A run records the values it actually used in its report, and an interrupted stage resumes from its newest intact checkpoint.
13 Results
These are the scores from the run that produced the file now installed as archer:14b. Base means the same model with the adapter switched off, on the same sentences, so the only thing that changed is the training.
| Job | Measure | Base | Tuned | How to read it |
|---|---|---|---|---|
| English to Cree | chrF++ | 8.24 | 11.60 | 41% more character overlap. Better, and still far from a fluent translation. |
| English to Cree | BLEU | 0.45 | 1.68 | The same story as chrF++. Low in absolute terms. |
| English to Cree | token F1 | 0.003 | 0.093 | Some of the right words now appear. Most still do not. |
| English to Cree | Cree words passing a spelling check | 36.8% | 92.4% | A phonotactic test, not a dictionary: it shows the shape of the language, not that every word is attested. |
| English to Cree | English left in the reply | 3.6% | 13.1% | Worse, and the table says so. Producing more Cree produced more leaks. |
| Cree to English | chrF++ | 11.96 | 19.12 | The easier direction, and the larger gain. |
| Cree to English | BLEU | 2.45 | 5.02 | Doubled. Use it to check a reading, not to publish one. |
| Cree to English | token F1 | 0.155 | 0.348 | Roughly twice the word overlap. |
| Song notes | token F1 | 0.708 | 0.793 | Answered from a section of notes held out of training. |
| Conversation | token F1 | 0.269 | 0.911 | 75 unseen phrasings about the project and the notes. |
| Abstention | correct refusals | 0 of 3 | 3 of 3 | Also 6 of 6 in the behaviour probe, with no false refusals. |
| Syllabics | letters in syllabics | 15.2% | 96.9% | Asked in English to write syllabics and it does. |
| Held-out Cree | perplexity | 12,147.9 | 96.2 | Lower is better. The largest change in the run. |
Run of 2026-11-02, adapter dpo, 120 sentence pairs the model never saw in both directions; 60 rows for the syllabics check; one section of an Archer note held out for the note question; three note questions it cannot answer and two it can. The numbers come from out/creebench_report.json and out/llm_tests.json.
The run writes this comparison itself and names where each bar came from. Every bar below is the base model, because that is the only earlier score measured on these sentences.
| Job | Measure | This model | Its base | Verdict |
|---|---|---|---|---|
| English to Cree | chrF++ | 11.600 | 8.241 | ahead |
| Cree to English | chrF++ | 19.121 | 11.956 | ahead |
| English to Cree | letters in SRO when SRO was asked | 99.1% | 98.5% | ahead |
| Song notes | token F1 | 0.793 | 0.708 | ahead |
| Conversation | token F1 | 0.911 | 0.269 | ahead |
| Abstention | accuracy | 1.000 | 0.000 | ahead |
| Syllabics | share of letters in syllabics | 0.969 | 0.152 | ahead |
| Held-out Cree | perplexity, lower is better | 96.2 | 12,147.9 | ahead |
Read the translation rows plainly. chrF++ 11.60 and BLEU 1.68 for English into Cree are low. The model has learned the shape of the language: 92.4% of its Cree words pass a Cree spelling check, against 36.8% before, and it now writes syllabics when asked. It has not learned to choose the exact word and ending a fluent speaker would choose. A translation from this model is a draft to hand to a speaker, not an answer to publish. The English leak rising to 13.1% is the clearest cost of that trade, and it is left in the table rather than hidden.
The strongest results are the ones with a right answer: refusing a question the notes cannot answer, writing the script that was asked for, and reading a note section it never saw. Conversation scoring 0.911 is real but narrower than it looks, because the 75 phrasings are all about this project and these notes.
14 Measurement
The benchmark runs the same held-out sentences through the base model with the adapters off and the tuned model with them on. The only thing that changes is the training, so the comparison is paired.
Translation is scored with chrF++ and BLEU, but also with three measures that matter more here. Validity asks whether every generated word exists in the lexicon or obeys Cree phonotactics, because an invented word is the failure a speaker notices first. Code-switch counts English leaking into a Cree answer. Syllabics checks whether the script is actually written, not just named.
The benchmark also measures the things a translation score cannot see. Does the same question get the same answer twice under greedy decoding? Does it still follow an English instruction after the Cree training? Does it decline a question the notes cannot answer, and does it also answer the ones they can? What does a Cree word cost in tokens now? Does its own confidence rise when its answer is better? Each of those is a separate probe with a fixed input list, so two runs of the same adapter produce the same numbers.
The evaluation cell writes its numbers and its charts into the run's export folder, in the same palette as this page. A number that changes in the code changes in the chart the next time the suite runs. Any Ollama model can also be scored on the identical prompts for a head-to-head, and this model's control tokens are stripped first, so the other model is judged on the instruction rather than on a prefix it has never seen.
| Check | The question it answers |
|---|---|
| Determinism | Does the same question get the same answer twice under greedy decoding? |
| Prompt ablation | What do the control tokens actually change? The same sentences are scored with and without them. |
| Paraphrase consistency | Do three ways of asking for the same translation land in the same place? |
| Abstention | Does it decline what the notes cannot answer, and does it still answer what they can? |
| English retention | Did the Cree training cost it the English it already knew? Perplexity is compared with the adapters on and off. |
| Tokenizer fertility | What does a Cree word cost in tokens now, against the base tokenizer? |
| Syllabics round trip | Can it write the script, and does converting it back to SRO give the sentence it started from? |
| Calibration | Does its own confidence rise when its answer is better, or is it confidently wrong? |
| Cost | Seconds per token and peak memory, measured rather than estimated. |
Every one of these is written to out/llm_tests.json and merged into the report, with the sample size beside it.
Lowercased words, counted as multisets, the same overlap the benchmark uses on a translation line. Punctuation and digits are not tokens.
15 Running it
The whole pipeline is a single Python file. It downloads the corpus, cleans it, builds the tokenizer, trains in three stages, benchmarks the result and exports a GGUF. Nothing in it needs Colab, Drive or a notebook.
This run used the t4 profile on one RTX 4090 with 24 GB. The corpus took about 15 minutes on its first build, continued pretraining 1 hour 32 minutes, instruction tuning 14 hours 10 minutes, preference tuning 24 minutes and the export 4 minutes. The same recipe on a smaller card runs more slowly; a 14B or 32B run needs a larger card and enough RAM to hold two copies of the model during the merge.
If the process stops, the same command continues it. Every finished stage is recorded against a fingerprint of its inputs and is skipped, and an interrupted training stage resumes from its newest intact checkpoint with the optimizer and the learning rate schedule. Changing the base model, the context length, the LoRA rank or the corpus invalidates exactly the stages that depended on it, so a new run cannot inherit numbers from an old one.
The export merges the adapter into a full precision copy, converts it with llama.cpp, quantises it to Q4_K_M and writes a Modelfile beside it. That file is what makes the model a single Ollama tag on the machine that runs the app.
The run writes out/creebench_report.json, out/llm_tests.json, a readable report_card.md, the plots, and an out/web/ folder with the compact JSON bundle and the SVG charts this page is built to use. Pointing the app at the result is one line in config/config.yaml: model: "archer:14b".
The file this page describes is archer-14b.Q4_K_M.gguf, about 9 GB at roughly 4.9 bits per weight, with its full SHA-256 in the archive's manifest.json. Testing it with ollama run sends no system prompt, and the persona and the evidence rule live in that system message. A bare prompt is answered by the base model's default assistant instead. Use the app, or call the same client the app uses:
16 Try it
Open the Archer Bot panel in the corner to ask a question, or use the retrieval explorer above to inspect the evidence first. The health endpoint reports whether the index built, how many chunks it holds, and whether a voice is configured, so a broken setup is visible rather than silent.
POST /api/chat · GET /api/chat/health · POST /api/chat/prompt · POST /api/chat/retrieve · POST /api/chat/reindex · POST /api/chat/sing · POST /api/chat/sing-from-lyrics
SingSong
SingSong takes a finished vocal and composes an instrumental around it. Not a backing track chosen from a library: a sequence generated to follow the performance. This is a paper-faithful implementation of Google Research and IRCAM's 2023 system, built from the token model up.
01 The idea
Generating ten seconds of 16 kHz audio sample by sample is a sequence of 160,000 decisions. Predicting a structured code sequence for the same ten seconds is a sequence of a few thousand. SingSong's central move is to make the second problem stand in for the first.
A neural audio codec turns the instrumental into discrete tokens at three time scales. A semantic encoder describes what the vocal is doing at 25 tokens per second. A coarse codec describes the body of the instrumental at 200 codes per second. A fine codec adds the top end at 400 codes per second. The model learns the conditional distribution of those codes given the vocal, and the codec turns the result back into sound.
The factorization is the architecture:
The first two factors are one T5-style encoder–decoder. It reads 250 semantic tokens of the vocal for a ten-second clip and emits 2,250 tokens: 250 semantic tokens for the instrumental, then 2,000 coarse tokens. The third factor is a decoder-only transformer that reads the 2,000 coarse tokens as a prefix and emits 4,000 fine tokens. Each stage can be trained, evaluated and sampled on its own.
02 The token layout
The vocabularies are not shared. Semantic, coarse and fine tokens each have their own range, and each position in the output has a mask that makes every token outside its range impossible. That is how one decoder can emit a structured sequence without a separate head for every stream.
At 16 kHz, ten seconds is 160,000 samples. The factorized sequence is 6,250 tokens for the same span, and it is the sequence the model is trained to predict.
03 The model
Stage 1+2 is a T5-style encoder–decoder trained with teacher forcing and cross-entropy on every target code. It reads the vocal's semantic tokens and writes the instrumental's semantic and coarse tokens. Stage 3 is a decoder-only transformer that reads the coarse prefix and writes the fine tokens.
Both stages use fixed sinusoidal positional encodings rather than T5's relative bias, pre-RMSNorm, gated GEGLU feed-forwards and tied output embeddings. Dropout is 0.1. Sampling uses a temperature of 0.85 with no top-k by default, and every invalid vocabulary subset is masked to negative infinity at each position. The base shape follows t5.1.1.base; the code supports the XL shape, though the published 3B run is a cluster job, not a laptop job.
The default long-form scheduler generates each ten-second window independently and crossfades with a complementary 50% overlap. The paper describes prompting the decoder with the previous five seconds; that path is implemented as an option for a checkpoint trained with the continuation objective, but it is out of distribution for a model trained on ten-second block layouts. The code states the choice rather than inventing a prompt and hoping.
04 The data
The published system used a very large private corpus. This implementation trains on what can be downloaded and licensed: MUSDB18-HQ for the general model, and a small, attributed Cree singing corpus for the adaptation.
Training pairs are vocal and instrumental stems, not mixtures. The pipeline accepts a stem layout directly, or a folder of mixtures with prepare --separate, which runs Demucs two-stem separation and refuses to train on a mixture as if it were an instrumental. Clips are filtered before tokenisation: an instrumental whose peak RMS is below −25 dBFS is dropped, and so is a pair whose vocal is 5 dB louder than its instrumental.
The paper's most important data choice is also implemented: white noise at σ = 0.01 is added to the vocal before feature extraction, at training and inference, and the vocal is represented only by semantic codes, never by its coarse codes. Separation artifacts in a “vocal” stem can otherwise teach the model to reconstruct the instrumental from the artifacts and then fail on a clean vocal. The noise and the semantic-only conditioning are the paper's fix, and this implementation follows it.
The Cree corpus is downloaded by tools/fetch_cree_songs.py, which records attribution, source URL, licence and split for every track. The Archer recordings on this site are not training data: they are mixed masters without confirmed training permission, and the repository does not treat a public listen as a training licence.
05 The contract
A token model can be subtly wrong for a long time: the loss goes down, the samples come out, and the vocabulary offsets are off by one. The test suite checks the contract directly, with large randomised samples rather than one hand-written clip.
| Check | Sample | Result |
|---|---|---|
| Vocabulary round trip | 750,000 token ids across all three ranges | 0 failures |
| Per-position masks | 270,000 positions | 0 violations |
| Stage 1+2 vocabulary | 5,122 ids | exact |
| Stage 1+2 positions | 2,250 per 10 s clip | exact |
| Fine vocabulary | 8,194 ids | exact |
| Fine positions | 4,000 per 10 s clip | exact |
The suite builds random sequences at every boundary, encodes and decodes them, and asserts that an invalid token can never be sampled at a position whose mask excludes it. The counts are the contract from the paper; the test is what keeps the implementation accurate about them.
06 Using it
Training is the deliverable. The code is the full recipe, and a run you own produces four artefacts: the codec, the semantic k-means, stage 1+2 and stage 3. Once those exist, the app exposes the renderer in the Live Demo's SingSong panel and the same page reports which stages are loaded.
The CLI covers the whole path: prepare-codec, train-codec, train-kmeans, prepare, train-stage12, train-fine, and render. status reports the state of every artefact and exits non-zero until a run is complete, which is the first command to run on a new machine. The training runbook, including VRAM expectations and resume behaviour, is in docs/singsong-training.md.
The panel lives in the Live Demo page, next to MySong. It accepts a vocal file and returns a mix, a band-only stem, or both. The status line above reports whether the trained artefacts and the Python environment are present on this machine.
07 The distinction
These are often confused because both are “AI music” features in the same app. They solve different problems. SingSong listens to a vocal and writes an instrumental that did not exist. RVC takes a performance that already exists and changes its timbre. The first is generation, the second is conversion, and neither one is a fallback for the other.
MySong
MySong reads a vocal melody, infers its key, chooses a chord sequence, and arranges a band around it. It began as a reimplementation of Microsoft's 2008 MySong and became a different decoder: flexible harmonic rhythm, a learned chord classifier, an exact semi-Markov search, and a band that knows more than one tradition.
01 The problem
Given only a tune, there are usually many chord sequences that fit. A good accompaniment is not the one with the most correct labels; it is the one that supports the phrase, changes when the phrase changes, and leaves room for the voice. That is a sequence problem, not a classification problem.
The 2008 paper framed it as a hidden Markov model: one node per measure, sixty chord states, a transition matrix learned from lead sheets, and a melody observation table. Viterbi finds the most likely path. It is elegant, fast, and it assumes one chord per bar, which is exactly the assumption most real songs break.
This implementation keeps the probabilistic framing and removes the grid. A chord can last 1, 2, 3, 4, 6, 8, 12 or 16 quarter notes, with a duration prior learned per meter and a separate prior for where in the bar changes happen. The decoder is a semi-Markov HMM, so a chord can begin on any beat. A truncated final chord is allowed, which means a 67-beat take does not have to pretend it is 68.
02 What changed
A fixed one-chord-per-bar grid cannot express a two-chord bar or a chord held for four bars. The semi-Markov decoder scores spans of eight possible durations, with priors learned per meter. The paper's fixed grid is still available as an evaluation baseline.
The paper's P(note | chord) table ignores note order and octave. This implementation fits a multinomial logistic regression on 52 features: metric chroma, downbeat chroma, onset chroma, tail chroma, log duration, bar phase and mode. It is exported to numpy, so inference needs no scikit-learn.
Krumhansl–Schmuckler correlation gets 55% exact key accuracy on held-out OpenEWLD, mostly confusing a key with its relative major or minor. A linear classifier trained on the corpus's key labels, transposition-augmented across all 24 keys, reaches 93.2% on the test split. It is a 24 × 12 matrix, not a model file.
A span classifier cannot see what came before it. A two-layer bidirectional GRU over per-beat features emits 60 chord log-probabilities per beat. It is trained with label smoothing and early stopping on a validation split carved out of the training songs. When its weights are present it is the observation model; otherwise the numpy classifier runs.
Viterbi over the semi-Markov model is exact. Forward–backward gives a posterior over chords for every span, so the panel can show alternatives and confidence. The posterior is tempered per beat before display because tagger errors are correlated; without that, every long chord reads as 100%. The mean chosen confidence is 0.62 against 0.53 core accuracy.
The first renderer was additive numpy: clean and plainly an oscillator. The default now renders each stem through FluidSynth and a General MIDI SoundFont, then applies balance, stereo width, convolution room, bus compression and a limiter. The numpy synth remains the deterministic fallback, and a MIDI file is always written.
03 The band
A Cree or Métis song should not have to sound like a pop demo before it can sound like itself. The band presets lead with the forms this project is for, and the Western styles sit after them as additional choices, not as the default frame.
A steady hand-drum pattern with bass and guitar support, no pad. The drum is the centre; the harmony stays out of its way.
Hand drum, rattles, boom-chuck guitar and bass. The fullest of the six, and the default for the SingSong panel's built-in band.
A denser percussion pattern and a higher intensity. The arrangement leans forward rather than settling under the voice.
A lighter percussion pattern with the jingle texture implied by the rhythm and the high end. The voice stays on top.
No drum kit; fiddle, guitar and bass. The melody instrument answers the vocal rather than doubling it.
Flute, light percussion, bass and pad. The quietest band, built for a voice that needs space around it.
After these six come pop, ballad, folk, jazz, waltz, ambient and rock. Every preset sets its own instrumentation, intensity, swing, humanisation and level. The style is a choice in the panel, not a guess made from the melody.
| Source | The takethe vocal as it went in, no band |
|---|---|
| Band | Hand drum, band onlyB major · 120.2 BPM · 3 segments |
| Mix | Take + bandthe band placed 16.2 dB under the voice |
Rendered by the same endpoint the upload panel uses. The readouts are the server's own headers, not a second analysis pass: key B, tempo 120.2, three chord spans, zero warnings.
04 Training
The base corpus is OpenEWLD, the public-domain subset of the Essen folksong collection: 502 songs, 52,417 melody notes, 19,569 chord spans. Splits are by song, 409 train, 49 dev, 44 test. The model is deterministic given a seed.
Five hundred lead sheets are enough to prove a decoder; making the observer strong takes more, which is why the large corpus exists. It is built from the 5,204-score AutoHarmonizer/Wikifonia archive, with two kinds of duplicate removed: normalised title and, more importantly, a transposition-invariant melody signature. Title matching alone missed 40 of the 44 test songs; the signature check removes them. The result is 4,571 extra scores with zero melody overlap against OpenEWLD train, dev or test.
The large tagger is trained by streaming one score at a time and keeping only the per-beat features. A naive load of five thousand songs plus the training graph exceeded 10 GB on a 27 GB laptop; the streaming path stays around 1.5 GB. The structural tables still come from OpenEWLD, which isolates the effect of the extra data and keeps the evaluation independent.
05 Results
Accuracy is frame-weighted at quarter-beat resolution against the lead sheet, at three levels: root, core (root plus one of the five triads) and full (exact extension). Key is exact key and mode; changes is F1 of chord-change positions within half a beat.
| System | root | core | full | key | changes |
|---|---|---|---|---|---|
| Two-tagger ensemble · shipped | 0.604 | 0.576 | 0.312 | 0.977 | 0.591 |
| Single 256-unit tagger | 0.592 | 0.560 | 0.315 | 0.955 | 0.600 |
| 128-unit tagger (ablation) | 0.544 | 0.522 | 0.299 | 0.932 | 0.552 |
| Fixed bar, large tagger | 0.584 | 0.559 | n/a | 1.000 | n/a |
| Template matching | 0.406 | 0.353 | n/a | 1.000 | n/a |
| Paper's generative observation | 0.249 | 0.206 | 0.146 | 1.000 | 0.600 |
44 held-out songs, 2026-10-27. Key is a 24-way classifier; the fixed-bar and template rows do not predict extensions, hence the dashes. The generative row uses the paper's P(note | chord) observation with the same transitions, durations and decoder, so the comparison isolates the observation model.
06 The mix
A generated band that sits at the same level as the singer is not an accompaniment. The renderer writes the band at one reference level and then places it under this particular take, so the balance does not depend on how loud the recording happened to be.
Across twelve renders the voice sat between 8.6 and 18.9 dB above the band, with a median of 13.0 dB. The spread is intentional: a sparse hand-drum arrangement can sit closer than a dense round-dance band without masking the voice. The loudest peak was 0.76 full scale, so the limiter is not working overtime to rescue a hot mix.
The panel is on the Live Demo page, in the same upload tool as SingSong. It reports the detected key, tempo and chord segments in the response headers, which is how the example above was labelled.
Measurements
This site makes a lot of numerical claims: cents, BPM, word error rate, root accuracy, latency. Each one comes from a reproducible test in tools/site_evidence.py, run against the real code, with the sample size printed beside the result. This page is the ledger.
01 Method
A single hand-picked example can make any system look good. The harness therefore prefers many randomised cases with known ground truth, and it records the sample size, the distribution and the failures, not just the mean.
Three kinds of test are used. Synthetic ground truth generates signals where the correct answer is known exactly: a note at 220 Hz, a pattern at 96 BPM, a chord progression with a written key. Corpus ground truth uses held-out songs and sentences with human annotations. Live endpoint tests call the running server and measure the response, which catches integration errors that unit tests cannot.
Each suite writes a plain JSON record and one or more SVG charts into frontend/assets/evidence/. The charts on the site are those files, not screenshots. A number that changes in the code changes in the chart the next time the suite runs.
| Suite | Sample | Headline |
|---|---|---|
| pitch | 236 tones + 21,600 smoother frames | median 3.7 cents, zero octave errors |
| notes | 60 melodies, 600 notes | F1 1.000, onset median 6.5 ms |
| rhythm | 96 patterns | onset F1 0.999, tempo median error 1.5 BPM |
| key | 120 songs | 57% exact, 71% relative |
| modes | 1,000 contexts | 0 invariant violations |
| voicing | 120 progressions | 0.90 vs 1.56 semitones movement |
| mysong | 44 held-out songs | root 0.604, key 0.977 |
| latency | 40 requests × 5 endpoints | 51 ms analyze, 53 ms sentiment |
| cree | 12,372 words + 1,000 FST | 0 round-trip failures |
| sentiment | 120 lines | 80% three-class accuracy, AUC 1.0 |
| vocables | 64 renders | 17.2% lower envelope error |
| rvc | 24 renders | 100% within 50 cents, 2.5 s median |
| asr | 40 utterances | 1.5% mean WER, 87.5% exact |
| bands | 12 renders | 13.0 dB voice over band, median |
| singsong | 750,000 tokens | 0 vocabulary or mask violations |
02 Pitch
236 synthesised tones sweep the singing range with vibrato, breath noise and level changes. The detector's answer is compared to the frequency it was given, in cents, and the error distribution is plotted rather than averaged into a single number.
03 Notes and rhythm
Pitch accuracy says nothing about segmentation. A detector can be perfectly in tune and still split one note into three. The note suite generates 60 melodies with 600 known notes and scores onset matching, pitch matching and F1 together.
04 Harmony
Three different questions, three different tests: does the key inference find the key, does the voicing search move smoothly, and does the selector obey its own rules under random conditions?
05 The generated band
The MySong suite is the largest corpus test on the site: 44 songs the model never saw, scored at quarter-beat resolution against the lead sheet. The decoder, transitions and split are shared with the baselines, so the comparison isolates the model.
06 Voice and language
Vocable pitch, vocable timbre, neural conversion and speech recognition each have a different failure mode, so each gets its own sample and its own chart.
07 Latency
Forty sequential requests per endpoint, after warm-up, on the same machine that serves the demo. Both the median and the 95th percentile are recorded, because a good median can hide a bad tail.
08 What a measurement changed
The raw detector's p90 error was 1,182 cents, almost entirely octave jumps. The jump-confirm smoother requires two consecutive frames before accepting a large leap, bringing p90 to 16 cents without adding latency to real note changes.
The original lookback covered six frames, about 140 ms, which is shorter than a musical phrase. Fitting a slope over a 0.6 s window, in log frequency, is what made the mode track shape instead of jitter.
Recomputing the drone from the latest note made it flip octaves mid-phrase and retrigger constantly. Locking it to the tonic, with hysteresis and a phrase-gap test, turned it into a held pedal.
The same vowel an octave down sounded muddy. A brighter, narrower vowel with a level trim that grows as the note descends is what makes the double read as a second singer.
A mixed ensemble gives the voice model several pitches at once. Per-stem conversion keeps every call monophonic and lets harmony stems rotate across voices.
Title-based deduplication left 40 of 44 test melodies in the training archive. A transposition-invariant signature check removed them, and the leaked score is not reported as a result. The clean number is 0.604.
09 Reproduce it
The heavy suites (rvc, asr, bands) talk to the running app, so start server.py first; they skip with a clear message if it is not reachable. The JSON record is written to reports/site_evidence.json, and the charts are written beside the site's own assets.
These are objective measurements: they show that the note is in tune, the band sits under the voice, and the timing lands on the grid. The demo pages are where you hear the result and judge the phrase for yourself.
Two live ways to use the system, plus two generative models, all on this page. Upload an MP3 and use the tab bar inside that panel to choose what happens to it: re-sing it through the synthetic voice model, generate a band with SingSong, or build a chord accompaniment with MySong. Or sing live into your microphone below. Pick any of the eight accompaniment modes from the harmony engine, same ones described on the Harmony page, and the robot sings back using a real formant-based vocal synthesizer instead of a plain tone. Then choose a voice texture (solo, duet, or a full concert choir) to change how many voices are actually singing that part.
Turn this off and the robot stays completely silent in both panels below, no matter which mode is selected. Same kill-switch behavior as the real engine.
A media-integrated performing artist, new media artist, filmmaker, writer, curator and educator. He has made solo performance works since 1984, and has performed professionally since 1982.
Archer Pechawis was born in Alert Bay, British Columbia in 1963. He has been a practising artist since 1984, with a particular interest in the intersection of Plains Cree culture and digital technology, merging traditional objects such as hand drums with digital video and audio sampling.
His work has been exhibited across Canada and in Paris, France, and featured in publications including Fuse Magazine and Canadian Theatre Review. He has received many Canada Council, British Columbia and Ontario Arts Council awards, won Best New Media Award at the 2007 imagineNATIVE Film and Media Arts Festival, and Best Experimental Short at imagineNATIVE in 2009.
Archer works extensively with Native youth as part of his art practice, teaching performance and digital media for various organisations and in the public school system. Of Cree and European ancestry, he is a member of Mistawasis First Nation, Saskatchewan.
The voice comes first: a hand drum, a song, a word held across a note
and then this system, answering without flattening it.
His songs work the way his performances do: a traditional object and a digital process set beside each other, each asking what the other is for. A hand drum carries a protocol. A sampler carries a protocol of its own. Almost all of the music treats that question as the material.
Cree vocables, the syllables a singer holds when the words step back, are the material this system works with. The singing stays Cree; the system follows it and stays out of its way.
This system is a new instrument for those songs: sing a line and it answers in a synthetic timbre, over the song's own instruments, on this machine and nowhere else. Nothing you record here leaves it.