Text-to-symbolic-to-audio music system¶

Udio-style UX with a Magenta-style core¶

Version: 1.0
Audience: product, research, infra, and audio engineering teams.

0. Scope, constraints, and licensing stance¶

Scope¶

Design a production-ready architecture where:

• Text + optional reference audio + optional MIDI →
  symbolic composition (Magenta-style) →
  full audio tracks (Udio-style UX).

Target:

• Full tracks (not just loops).
• Non-musicians: text-only UX.
• Musicians/producers: can supply MIDI, stems, motifs.

Licensing constraints¶

Everything below is meant to be implementable with commercially compatible components:

• Code: only use Apache-2.0, MIT, BSD or equivalent; no non-commercial weights.
• Datasets:
• Public-domain scores/audio and derived MIDI.
• CC-BY / CC-BY-SA / similar, filtered to exclude “NC”.
• Licensed commercial catalogs with explicit training and commercial-reuse rights.
• User-provided content under explicit opt-in terms.
• Explicitly excluded for training:
• CC-BY-NC datasets (e.g., MAESTRO).
• Pretrained models that are CC-BY-NC or otherwise non-commercial.
• Ambiguous datasets like Lakh/Slakh for production training, unless legal signs off.

Architectures and pipelines are defined so they do not rely on non-commercial-only assets.

1. Objectives¶

1. Define an end-to-end text→symbolic→audio pipeline.
2. Compare this against pure audio-generation systems.
3. Identify capabilities unlocked by symbolic control.
4. Propose 1–3 detailed, buildable architectures.
5. Highlight riskiest assumptions and minimal experiments before any large build.

2. End-to-end pipeline (conceptual backbone)¶

All architectures share this backbone.

2.1 User inputs¶

• Text prompt(s); optional “negative prompt” (“no drums”, “no vocals”).
• Optional reference audio (tracks, stems, or short clips).
• Optional symbolic inputs:
• MIDI motifs, chord charts.
• Structural markers (“this is verse, this is chorus”).
• Existing stems to match or complement.

2.2 Intent encoder¶

Outputs a music intent representation, split into:

• Text encoder:
• Semantic embedding.

• Parsed tags: genre, mood, tempo range, meter hints, energy shape, length target, instrumentation hints.

• Audio encoder (for reference audio):

• Style embedding (timbre/mix/groove).

• Optional extracted features:

  • Beat/tempo.
  • Harmonic profile (rough chords/key).
  • Groove pattern (drum onsets, swing, syncopation).
  • Section boundaries.

• Optional symbolic encoder:

• For user-provided MIDI motifs or chord progressions.
• Outputs motif embeddings and hard constraints (e.g., chord grid).

2.3 Symbolic planner (global structure)¶

Produces the “plan”:

• Global metadata:
• Overall length (bars / seconds).
• Tempo map, meter changes.

• Key/scale over time.

• Section sequence:

• Section types: INTRO, VERSE1, PRECHORUS, CHORUS, BRIDGE, OUTRO, etc.
• Section lengths in bars.

• Section-level attributes:

  • Target energy level.
  • Texture density (sparse/dense).
  • Instrumentation hints.

• Chord grid and basic register:

• Chord per bar or finer.
• Rough register (low/mid/high emphasis).
• “Hook bars” or motif anchor points.

2.4 Multi-track symbolic composer¶

For each section and role:

• Roles: DRUMS, BASS, CHORDS, LEAD, PAD, ARP, FX, VOCAL_MELODY, etc.
• Inputs:
• Section plan.
• Chord grid.
• Intent embeddings (style, mood, groove).
• User seeds (MIDI, locked bars).
• Outputs:
• Event-based sequences (per role) in a shared time grid.

2.5 Performance renderer¶

Adds expressive performance details:

• Micro-timing (humanization).
• Velocity curves and dynamics.
• Articulations (legato/staccato/etc.).
• Controllers (mod wheel, expression, pedal).

2.6 Instrument and production engine¶

Maps symbolic performance to audio:

• Instrument selection per role (sample-based, NN synth, etc.).
• FX chains per track and bus.
• Panning and spatialization.
• Mixing and mastering.

2.7 UX layer¶

Two user profiles:

• Non-musicians:
• Simple prompt + sliders/toggles:
  • Length.
  • Complexity.
  • Repetition.
  • Energy shape.
  • “With/without vocals”, “with/without drums”, etc.

• Actions:

  • “Regenerate song”, “Regenerate chorus only”, “Change style to X”.

• Musicians / producers:

• Can upload:
  • MIDI motifs/chords.
  • Stems/reference audio.
• Can:
  • Lock/unlock sections or roles.
  • Regenerate per-part or per-section.
  • Export stems/MIDI/full mix.
  • Edit MIDI directly and re-render.

3. Representations and conditioning¶

3.1 Symbolic representation¶

Use a hierarchical, event-based representation:

• Global / section level
• Section tokens: SECTION_START(type, length_bars), SECTION_END.
• Key/scale, tempo, meter.

• Section attributes: energy, density, instrumentation tags.

• Track level

• Track header: TRACK(role, instrument_family, id).
• Events:
  • TIME_SHIFT(Δt).
  • NOTE_ON(pitch, velocity), NOTE_OFF(pitch).
  • CONTROL_CHANGE(controller, value).
  • PROGRAM_CHANGE(instrument_id).
  • Optional: articulation events (ARTICULATION(legato), etc.).

• All events are aligned to (bar, beat, tick) for deterministic DAW mapping.

• Structure graph (optional)

• Motif nodes with IDs (A, B, C).
• References from bars/sections to motif IDs.

3.2 Style and control signals¶

Break conditioning into:

1. Global style embedding
2. From text + reference audio.

3. Encodes:

   • Genre/era.
   • Energy and dynamics profile.
   • Production aesthetic (clean/lo-fi/vintage/modern).
   • Instrumentation tendencies.

4. Aspect-specific attributes

5. Derived via parsing and/or small models:

   • Tempo/meter: from phrases like “slow ¾ waltz”, “fast ⅞”.
   • Energy curve: “starts minimal, huge final chorus”.
   • Groove: “straight”, “swing”, “half-time”, “syncopated”.
   • Harmony hints: “jazzy ii-v-i”, “standard pop four-chord”, “modal”.

6. Discrete tags

7. Genre, instrumentation, era.
8. Vocal presence, language hints, lyric topics (if generating lyrics).

These are fed into section planner and track composers as context, and into the production engine for timbre/mix choices.

4. Symbolic composition core¶

4.1 Model decomposition¶

Two main design options:

1. Hierarchical multi-stage core (recommended)
2. Section planner: predicts form and chord grid.
3. Role-specific composers:
   • Separate models per role (drums, bass, chords, lead, textures, vocals).

4. Pros:

   • Easier per-role control and partial regeneration.
   • Specialization by instrument role.

5. Single large multi-track model

6. Shared transformer over a unified token stream with role/section tokens.
7. Pros:
   • Single model, simpler infra.
8. Cons:
   • Harder to isolate roles; partial regen harder.

Given UX requirements, we favor hierarchical + per-role.

4.2 Section planner (SP)¶

Inputs:

• Text and style embeddings.
• Optional:
• User-specified form (“ABABCB”).
• User chord charts.
• Reference audio features.

Outputs:

• Section sequence with lengths.
• Chord progression at bar/sub-bar resolution.
• Global and section-specific attributes:
• Energy curve.
• Density.
• Register emphasis.

Model type:

• Transformer or diffusion over symbolic “plan tokens”.
• Trained on aligned chord+form datasets (public domain, licensed).

4.3 Role-specific composers¶

Per-role models, e.g.:

• Drums model:
• Conditioned on section type, tempo, groove, style embedding.
• Outputs kick/snare/hihat/tom/percussion events.

• Also outputs micro-timing and velocity patterns or a latent that a performance model uses.

• Bass model:

• Conditioned on chords, drums, style.

• Emphasizes rhythm locking with kick and functional harmony.

• Harmony model (chords/pads):

• Expands chord grid into voicings and textures per section.

• Takes density and energy as additional controls.

• Lead/melody model:

• Conditioned on chords, style, section.
• Can condition on user motif seeds.

• For vocal melody, also conditions on lyrics syllable/phoneme sequences (see vocals section below).

• Texture/FX model:

• Generates sparse events (risers, impacts, sweeps, fills).

Model family:

• Transformer with relative attention for long sequences.
• Optional latent (VAE/diffusion) for interpolation and variation.

4.4 Continuation, interpolation, variation¶

• Continuation:
• Input: text/style + existing symbolic for first N bars.

• Lock existing sections; SP generates further sections; composers generate new content from the last bar onward.

• Interpolation:

• Encode sections into latent space (e.g., MusicVAE-like).
• Interpolate between two states and decode under current style.

• Use for “between these two versions” operations.

• Variation:

• Sample alternate latents near existing one.
• Regenerate with same constraints for controlled diversity:
  • “Same chords, different melody.”
  • “Same groove, more fills.”

4.5 Enforcing global structure¶

Mechanisms:

• Explicit section tokens and positions (bar index within section).
• Loss components:
• Penalize deviations from chord grid.
• Encourage energy pattern adherence per section.
• Constraints:
• Keep key changes and modulations limited unless requested.
• Repetition control to ensure hooks recur at desired positions.

5. Instrument and production layer¶

5.1 Protocol: symbolic → instruments¶

Use MIDI + metadata:

• Per-track:
• MIDI events.
• Role label.
• Instrument family tag (piano, synth lead, acoustic guitar, etc.).

• Articulation tags and performance directives:

  • Legato/staccato.
  • Dynamics bands.
  • Humanization bounds.

• Global:

• Target mix profile (genre-specific).
• Loudness target.

This is compatible with DAWs and external instruments.

5.2 Production subtasks¶

1. Timbre selection
2. Map track roles + style → instrument presets.

3. Banks:

   • Sample-based instruments (soundfonts, libraries).
   • DDSP-like neural instruments for expressive control.

4. Sound design

5. For electronic genres, parameterized synth templates:

   • Oscillators, filters, envelopes, modulation.

6. FX chains

7. Track-level: EQ, compression, saturation, delay, reverb.
8. Bus-level: drum bus, instrument bus, vocal bus.

9. Master bus: limiting, subtle EQ.

10. Spatialization

11. Panning per track.

12. Simple stereo or binaural enhancements.

13. Mixing

14. Rule-based starter:
    • Role-specific starting levels and pans.
15. Optional learned mixer:
    • Inputs: track features (RMS, spectral centroid, role, style).
    • Outputs: fader positions, send levels, basic EQ/comp choices.

5.3 Production engine variants¶

Variant 1: Classical sampler + rules

• Simple, fast, robust.
• Easiest to implement.
• Good for first production version; audio quality is acceptable but not state-of-the-art.

Variant 2: Score-conditioned neural synthesizer (MIDI→audio)

• Per-instrument MIDI→audio models (e.g., MIDI-DDSP style).
• Higher realism, especially for expressive instruments (strings, winds).

Variant 3: Score-conditioned diffusion / codec LM

• Directly generate audio (stems or mix) conditioned on the symbolic score and style.
• Highest potential audio quality; highest training and inference cost.

Variant 4: Hybrid (Symbolic → rough audio → refinement)

• Render with sampler or neural synth.
• Refine with an audio→audio diffusion model that improves timbre and mix while preserving structure.
• Good compromise between data complexity and quality.

5.4 Latency impact¶

• Sampler / MIDI-DDSP: low to moderate latency, suitable for interactive iterations.
• Score-conditioned diffusion: high latency; suitable for offline “render at high quality”.
• Refinement diffusion: moderate to high latency, depending on model and clip length.

Strategy:

• Use fast sampler/NN synth for immediate preview.
• Offer offline “HQ render” using diffusion/refinement.

6. UX and controllability¶

6.1 Non-musician UX¶

Expose symbolic power through prompt patterns and basic controls:

• Prompt examples:
• “Slow ¾ sad piano ballad, no drums, 2 minutes, big ending.”
• “Aggressive 140 BPM trap beat, heavy 808s, dark pads.”
• UI controls:
• Length slider (e.g., 15 s – 4 min).
• Complexity slider (simple ↔ dense).
• Repetition slider (more hook ↔ more variation).
• Energy shape presets (“flat”, “build-up”, “big chorus”, “fade-out”).
• Toggles: with/without vocals, with/without drums, instrumental only.

System maps these into structure plan and composer constraints.

6.2 Power user UX¶

Additional controls:

• MIDI upload:
• Motif as main hook, bass pattern, or chord progression.
• Locking:
• Lock sections; regenerate only others.
• Lock roles; regen only drums/harmony/lead/etc.
• Section-level controls:
• Force length in bars.
• Key changes, modulations.
• Explicit repetition of motifs (“reuse motif A here”).

6.3 Edit propagation¶

Mechanism:

• User edits MIDI in a DAW-like view.
• System treats edited bars as hard constraints.
• Supports scoped regeneration:
• Strict local: regenerate only selected bars for a role.
• Halo local: regenerate a small neighborhood (e.g., ±4 bars).
• Global: re-plan later sections while preserving “landmark” bars (hooks, cadences).

Audio rendering:

• Re-render only affected parts.
• Reuse stems elsewhere to limit compute and preserve continuity.

6.4 DAW and live integration¶

• Export:
• Full mix (WAV), stems, track-level MIDI, and project metadata (JSON with structure).
• Import:
• User stems (for style reference or as fixed content).
• User MIDI motifs and chord charts.
• Live tools:
• API for loopers/controllers to:
  • Request 8-bar continuation.
  • Swap out drums or bass while maintaining harmony.

7. Data, training, personalization¶

7.1 Dataset requirements¶

1. Symbolic composition
2. MIDI/score datasets:
   • Public domain and CC-BY/SA scores (classical, jazz, traditional).
   • Licensed modern MIDI collections for pop/EDM/hip-hop.

3. Annotations:

   • Form (section labels).
   • Chords, keys, tempo maps.

4. Style conditioning (text↔music)

5. Corpus of tracks with:
   • Text descriptions (genre, mood, instrumentation, usage).
   • Optionally: generated captions refined by humans.

6. Coverage of diverse genres and moods.

7. Production modeling (symbolic↔audio)

8. Score+audio datasets:
   • Public-domain/CC-BY classical recordings with aligned scores.
   • In-house recorded sessions with scores and stems.

9. Synthetic datasets:

   • Symbolic corpora rendered through known virtual instruments to create aligned MIDI↔audio pairs.

10. Vocals-specific data

11. Singing datasets with lyrics, phoneme alignments, and melody.
12. Multilingual where possible.
13. Must be licensed for training and commercial use.

7.2 Dataset governance¶

• Dataset registry:
• For each dataset:
  • Source, license, allowed uses, attribution requirements.
• Filters:
• Exclude NC and unclear-license content.
• Versioning:
• Data packs with explicit version IDs.
• Models track which packs were used.

7.3 Personalization (e.g., per artist)¶

Goal: “Stillith-style” output without copying.

Approach:

• Base models trained on broad corpus.
• Per-artist adapters:
• Low-rank adapters, style heads, or additional conditioning tokens.
• Training adapters:
• Only on the artist’s licensed catalog.
• Strong augmentation (key shifts, timing perturbations, phrase-level recombination).
• Safety:
• Similarity checks against artist’s own catalog to avoid near-duplicates.
• Artist can opt-out or constrain usage.

8. Evaluation¶

8.1 Layer-wise metrics¶

Symbolic layer

• Tonal consistency:
• Key and chord adherence.
• Phrase structure:
• Distribution of phrase lengths and boundaries.
• Repetition vs novelty:
• Motif detection and reuse rate.
• Rhythmic stability:
• Drums and bass onset patterns vs expected styles.

Audio layer

• Objective metrics:
• Signal quality (no clipping, noise).
• Perceptual distances (e.g., FAD-like metrics) compared to suitable corpora.
• Stem quality:
• Separation metrics if training with stems.

UX

• Time to first “kept” track.
• Number of regenerations before user export.
• Perceived control:
• Likert scores on “Does the system do what I asked?”.
• Edit efficiency:
• How many steps to reach user satisfaction starting from a prompt.

8.2 Human evaluations¶

Types:

1. Audio quality vs musical coherence
2. Listeners rate:
   • Audio fidelity.
   • Musical structure (form, motifs, harmonic sense).

3. Compare symbolic→audio vs pure audio baselines on same prompts.

4. Control evaluations

5. Ask users to request specific edits:
   • Shorter intro.
   • Same chorus, different verses.
   • Change drums, keep everything else.

6. Assess whether outputs satisfy constraints while preserving desired content.

7. Expert panels

8. Producers/composers evaluate:
   • Suitability for real projects.
   • Editability.
   • Reliability of per-part regeneration.

8.3 Baselines¶

• Pure audio text-to-music models (e.g., MusicGen-/MusicLM-style, StableAudio-like).
• Pure symbolic models (Magenta-style) rendered through basic soundfonts.
• Human benchmarks (e.g., simple library music / production cues for similar tasks).

9. Legal, ethical, safety¶

9.1 Logging and transparency¶

• Log:
• Prompts, seeds, model versions, generated content fingerprints.
• Whether user content is allowed for training.
• Provide:
• Per-user data export (prompts, outputs, flags).
• Controls:
  • “Allow my content for training”: yes/no.

9.2 IP risk mitigation¶

• Similarity checks:
• Melody:
  • Interval-based contour and n-gram analysis.
• Rhythm:
  • Pattern similarity for drums/bass.
• Harmony:
  • Chord sequence overlap.
• Audio:
  • Fingerprinting for near-duplicate detection.
• Policy:
• Conservative thresholds; reject or alter outputs that are too close to training songs or known external works.
• Clear messaging when content is blocked.

9.3 Content safety and misuse¶

• Prompt filtering:
• Disallow or moderate prompts that request disallowed or harmful content.
• Lyrics guardrails:
• Text moderation on lyric generation and user-provided lyrics if requested for vocals.
• Voice/artist impersonation:
• Artist-style adapters are opt-in and restricted to rights holders.
• Block explicit impersonation requests for third-party public figures or artists.

9.4 Data lineage and removals¶

• Data packs:
• Track which datasets feed which models.
• Right-to-remove:
• If a pack is revoked, retrain or fine-tune excluding that pack when needed.

10. System-level considerations¶

10.1 Scalability¶

• Microservices:
• Intent encoding.
• Symbolic planning/composition (stateless).
• Rendering (audio).

• Storage for symbolic scores and stems.

• Long tracks (5–10 minutes):

• Generate per section with overlapping context.
• Store section-level scores and stems for partial regeneration.

10.2 Caching and reuse¶

• Cache:
• Text→intent embeddings.
• Intent→structure plans.
• Per-track stems.
• On “iterate on my track”:
• Reuse upstream results.
• Only recompute changed roles/sections.

10.3 Model versioning and reproducibility¶

• Tag every generation with:
• Model versions: text_enc_vX, symbolic_core_vY, renderer_vZ.
• Seeds and randomization settings.
• Guarantee:
• Old tracks can be reproduced with pinned versions.
• Upgrades require explicit opt-in.

10.4 Latency and UX¶

• Plan for “progressive refinement”:
• Quick low-quality preview (few seconds).
• Optional HQ render later using heavier models.
• Provide:
• Progress indicators for long renders.
• Section-based preview playback while later sections render.

11. Three concrete architectures¶

11.1 Architecture A: Symbolic-first + neural MIDI synth¶

Goal: Maximize symbolic control, moderate-to-high audio quality, low complexity.

A.1 Flow¶

Prompt / audio / MIDI
→ Intent encoder
→ Section planner
→ Role-specific symbolic composers
→ Performance renderer
→ Instrument mapper + sampler / MIDI-DDSP
→ FX + mixing + mastering
→ Audio + stems + MIDI

A.2 Characteristics¶

• Symbolic core:
• Section planner + per-role transformers.
• Renderer:
• v1: high-quality sample libraries or open soundfonts.
• v2: MIDI-to-audio neural instruments for realism.

Pros

• Excellent controllability and editability.
• Lower data complexity and compute cost.
• Easy to integrate with DAWs and live workflows.

Cons

• Audio quality ceiling lower than best diffusion-based systems.
• Harder to match highly detailed, trendy production styles.
• Needs good orchestration and mixing heuristics to sound competitive.

11.2 Architecture B: Symbolic + score-conditioned diffusion¶

Goal: Combine symbolic control with very high audio quality.

B.1 Flow¶

Same symbolic stack as A, then:

Symbolic score (multi-track, expressive)
→ Score encoder
+ Style embedding
→ Score-conditioned diffusion / codec LM
→ Stems or full mix
→ Mastering

B.2 Characteristics¶

• Score representation:
• Multi-channel piano-roll or event grid encoding.
• Condition:
• Score for structure.
• Style embedding for timbre/mix.
• Training:
• Needs aligned score+audio and/or robust synthetic training.

Pros

• Strong structure preservation via explicit score.
• High-quality, stylistically rich audio.
• Can re-render with different styles while keeping composition fixed.

Cons

• Requires substantial aligned score+audio data.
• High training and inference cost.
• Alignment complexity is significant, especially for multitrack modern genres.

11.3 Architecture C: Rough audio → refinement diffusion¶

Goal: Leverage symbolic control and a cheap renderer, then add realism via refining diffusion.

C.1 Flow¶

Same symbolic stack as A, then:

Symbolic performance
→ Fast sampler / MIDI-DDSP renderer (rough stems/mix)
+ Style embedding
→ Audio-to-audio diffusion refiner
→ Final mix

C.2 Characteristics¶

• Diffusion learns to map “rough” to “polished” while preserving:
• Timing.
• Harmony.
• General structure.
• Training:
• Synthetic pairs: (rough render, higher-quality reference) or self-augmentation.

Pros

• Less dependent on fully aligned score+audio data.
• Structural control anchored by symbolic and rough renderer.
• Can iterate audio-quality improvements separately from symbolic core.

Cons

• Risk of altering structure if refiner is too aggressive.
• Risk of style homogenization.
• Two-stage pipeline can be harder to debug.

12. Cross-architecture trade-offs¶

Pragmatic strategy:

• Build Architecture A first as the production baseline.
• Run Architecture C as medium-horizon R&D (refinement) once A is stable.
• Treat Architecture B as long-horizon, data-gated R&D starting with single-instrument domains.

13. Self-assessment and loopholes¶

This section critiques the design and adds refinements.

13.1 Vocals and lyrics are under-specified¶

Gap:

• No detailed plan for:
• Text→lyrics.
• Lyrics→melody alignment.
• Phoneme-level singing synthesis.
• Multilingual support.

Impact:

• For many users, “song” implies vocals.
• Without a vocal pipeline, the system is mainly an instrumental generator.

Needed sub-system:

1. Lyrics generator (optional)
2. Text prompt → lyrics, or user-provided lyrics.

3. Control: language, theme, syllable count per line, rhyme scheme.

4. Lyrics-to-melody model

5. Input: lyrics (words, syllables, phonemes), section plan, style.

6. Output: vocal melody line with:

   • Pitch and duration per syllable.
   • Phrasing and pauses.

7. Singing voice synthesis

8. Input: phoneme sequence, durations, F0 curve, style/voice embedding.

9. Output: vocal audio stem.

10. Integration

11. Architecture A: treat vocals as another role rendered via a neural singing synthesizer.
12. B/C: include vocal conditioning in score/audio models.

This is a high-risk, high-impact vertical that should be planned separately.

13.2 Reference audio conditioning is under-specified¶

Gap:

• Vague notion of “style embedding + harmonic/groove summary.”
• No concrete description of:
• Beat/tempo detection.
• Harmony and groove extraction.
• Section boundary detection.
• Confidence management.

Impact:

• Weak inference from reference audio → mismatched chords/groove.
• Important for Udio-style “make something like this” UX.

Required components:

• Beat and tempo estimator.
• Key and chord estimator.
• Drum/groove analyzer.
• Section boundary detector.
• Confidence scores and strategies for:
• Hard constraints (high-confidence features).
• Soft hints (low-confidence features).

13.3 Data realism for modern genres is underplayed¶

Gap:

• Assumes “public-domain classical + some licensed MIDI” will cover:
• Pop, hip-hop, EDM, modern production.

Reality:

• High-quality, multi-track, modern-genre datasets with clear training rights are scarce and expensive.
• Architectures B/C in particular are data-hungry.

Implication:

• Architecture A remains the most realistic path to a usable system in the near term.
• B/C depend on building or licensing substantial modern-genre, multitrack data.

13.4 Score–audio alignment complexity (Architecture B)¶

Gap:

• Assumes we can get strongly aligned score+audio data for multitrack pop.

Reality:

• Existing aligned datasets are mostly:
• Single-instrument (piano).
• Small ensembles.
• For multitrack popular music, alignment is nontrivial and often unavailable.

Implication:

• Architecture B should start with narrower domains:
• Solo piano, small ensembles.
• Full multitrack score-conditioned diffusion is a long-term bet.

13.5 IP similarity and content safety are simplified¶

Gap:

• Only simple “melody overlap” checks mentioned.
• No multi-axis similarity or detailed moderation plan.

Needed improvements:

• Multi-axis similarity detection:
• Melody contour-based, not just exact sequences.
• Rhythm pattern similarity.
• Chord progression similarity.
• Audio fingerprinting as backstop.
• Content moderation:
• Prompt and lyric filtering.
• Guardrails on explicit/harmful content.
• Governance for artist impersonation and style cloning.

13.6 Edit locality and renderer–composer mismatch¶

Gaps:

• Edit propagation:
• Local edits can have global implications, especially for groove and key.
• Renderer–composer mismatch:
• Symbolic models might generate notes outside instrument ranges or dense textures that render poorly.

Mitigations:

• Define explicit edit scopes (strict local, halo local, global).
• Integrate renderer constraints into training:
• Range-aware losses.
• Density penalties.
• Post-processing (fold notes into range, thin textures).

13.7 Latency, previews, and UX reality¶

Gap:

• Latency discussion is qualitative.

Reality:

• Users expect:
• Quick feedback for exploration.
• Tolerate slower HQ renders if they opt in.

Mitigation:

• Design for:
• Fast preview (Architecture A sampler).
• Optional HQ render (B or C) with progress indicators.
• Possibly stream sections as they are ready.

13.8 Internationalization and diversity¶

Gap:

• Design is implicitly Western/English-centric.

Needed:

• Multilingual prompt and lyrics support.
• Non-Western scales and rhythm systems in symbolic models.
• Dataset diversity to avoid collapse into narrow genre/region.

14. Revised risk view and experiment priorities¶

14.1 Architecture A (Symbolic + synth / MIDI-DDSP)¶

Main risks:

• No or weak vocal support.
• Gap vs top-tier timbral realism.
• Composer–renderer mismatch for complex material.

Key experiments:

1. Text→form+chord→multi-track prototype:
2. Instrumental, no vocals.

3. Evaluate structural coherence and editability.

4. Renderer capability study:

5. Identify instrument ranges, density limits; adjust symbolic training accordingly.

6. Early vocal stub:

7. Simple vocal melody generation and naive singing synthesis.
8. Goal: surface integration issues early.

14.2 Architecture B (Score-conditioned diffusion)¶

Main risks:

• Data acquisition and alignment for multitrack.
• Training and serving cost.

Key experiments:

1. Single-instrument piano score→audio:

2. Focus on structure preservation and style variation.

3. Tiny multitrack pilot:

4. In-house recorded band with full scores and stems.
5. Train a toy model to see if multitrack score-conditioning is feasible.

14.3 Architecture C (Rough→Refine diffusion)¶

Main risks:

• Structure drift (timing, pitch) in refinement.
• Style homogenization.

Key experiments:

1. Structure preservation test:
2. Rough renders with known symbolic structure → refine.

3. Measure F0 and onset changes.

4. Style generalization test:

5. Train refiner on limited genres; test on other styles.
6. Evaluate whether it “pulls” outputs into a narrow house sound.

15. Bottom line¶

• A symbolic→audio architecture gives:
• Strong control over form, motifs, harmony, and per-part regeneration.
• Better editability and reproducibility than pure audio models.
• A pure audio model may win in timbral realism alone but lacks:
• Precise structural control.
• Reusable symbolic artifacts (MIDI, chords, motifs).

In practice:

• Build Architecture A as the main product backbone.
• Treat vocal support, refinement diffusion, and score-conditioned audio as separate but connected research tracks.
• Gate large investments on early experiments and real usage metrics, not only on offline lab scores.

This document is ready as an internal reference for planning, design reviews, and early implementation discussions.