Ask HN: Are there AI models for generating sounds based on a text and reference?

34 points by onemiketwelve · 18 comments
I've been having a hard time finding a solution. Is there really no commercialized model that I can feed in a reference sound and text instruction and get another sound out?

Right now having a multimodal inputs to image or text output is a commodotized, solved problem. IE you can put a prompt for some image and use a reference image to guide the model on what you want. After all, a picture is worth a thousand words right?

Ive also used a text and audio input in order to get a text description or classification out.

I cannot for the life of me find a solution for Audio + text -> Audio

My usecase is that I'm trying to generate new sound effects based on a source that doesn't have that many clean examples and thought this would be just another solved workflow but I'm not finding much. Tried elevenlabs SFX but it's only text->audio and without the reference, it's hard to guide with just text and recreate a sound with any accuracy. I tried the Stable Audio 3 model which is supposed to do what I want but the results were terrible. This modality seems to be stuck where image generation was in 2016. Is there something else or is this just not a common need?

18 comments

soundworlds
You should look into using AI to generate code that synthesizes sounds. Not exactly what you asked for, but I do think it is an approach worth considering: https://m.youtube.com/watch?v=1-i45X5aj94
SyneRyder
I've had some luck with that approach - giving a frontier LLM a reference sound, asking it to replicate the sound via code (basically asking it to create a physical model, I guess) and being able to steer the output by talking with the LLM.

I was working with very simple elements, but I was surprised by some of the outputs.

I would have also suggested Waves Illugen, but it turns out that is text-only, you can't give it an audio reference.

dmos62
Great video!
thangalin
https://github.com/OpenMOSS/MOSS-TTS

KeenLore is my locally hosted full-cast emotive audiobook generator I'm developing for my novel:

https://www.youtube.com/watch?v=WAeHgE94rVo

Another HN user suggested embedding sound effects. The lead author of MOSS-TTS emailed me that the next version will have a SOTA voice designer. Poke around the voice design space, you'll find text-to-effects here and there. I certainly have my eyes on this space.

ninininino
KeenLore is really cool.
moonu
Even though they're technically trained for music, it might be worth testing Suno/Lyria to see if they're able to do this. You might have to isolate it afterwards, but seems like it could be viable
xg15
Maybe a dumb idea, but how good are image models with manipulating spectrogram images? Then a workaround could be: convert input audio to spectrogram -> pass spectrogram + prompt to an image model -> convert modified spectrogram back to audio waveform.
Buttons840
I wouldn't call it a dumb idea, but there's soooo much subtlety to sound that wont be visible in any reasonably sized image.
narrationbox
Could be interesting, though spectrogram voice gen is a step back in terms of advancement. Vocoder models that take in Mel Specs suffer from a lot of issues like hissing due to Griffin Lim issues. Would be curious to see if diffusion models can work on neural codecs directly.

I think Google had one called riffusion (the first version was designed for specs)

jajazheng
Hi! I’m unsure if this is what you were looking for, QWEN3 TTS has a “clone” feature. You can use the local version, give it a voice reference and text and it will read the text with your instructions with the sample voice you selected. I used this combo to make a customised “audiobook” for my mother. It worked quite well.
narrationbox
Plenty of models take in text + audio and spits out audio. It's the format of most newer generation accent conversion/voice cloning models.

What's your exact use case?

chr15m
Sound effects are completely different to voice, which those models are trained to output.
bobosha
Not exactly what you asked for, but are you aware of CLAP? https://github.com/LAION-AI/CLAP

This generates audio embeddings - much like CLIP does for visual inputs.

jallmann
Daydream Music - https://daydream.live

The DEMON realtime engine is open source. The hosted service has a number of integrations with popular audio tools. (I'm on the team)

brudgers
Sometimes doing the work is easier than shopping for a tool that does the work for us.

Particularly when the application is a corner case or requires aesthetic judgment.

Of course in the case of sound effects shopping is a well established practice. Sound libraries and foley artists are widely available.

Finally, sound is much harder than images because psychoacoustics are more complex than the mechanical descriptions while human visual experience is a massively filtered set of stimuli.

We chunk the visible into symbols/archetypes like chairs and trees to a much much greater degree than we chunk sounds into symbols.

What is the sound of a chair? Of a tree?

LLM music is heavily dependent on genre. There is no genre for SFX. Good luck.

chr15m
Apparently the AudioX and AudioLDM(2) models do this but I think you've found a genuine gap.