Astra Taught Qwen TTS My Voice

If you are reading this on X or you don't live under a rock by now you have probably heard about Sparks of Weight-Space Programming. If you have then feel free to skip the first section on Weight Space Programming and go straight to the voice experiment.

I wanted to see whether this could work for voice cloning. So I gave Astra my voice samples and asked it to make a TTS sound like me by editing the weights directly.

TL;DR

• Started with Kokoro 82M, Astra could change its weights but It sounded like another person.
• For Qwen, Astra found two particular heads, one to increase the speaker contribution and the other to change the acoustic predictions.
• The generated voice was so close to mine that my father and a few of my friends couldn't tell it was cloned.
• Although the edits were sounding like me in English, they didn't work for Hindi/Urdu. So, I asked Astra to keep trying and after a few attempts it decided the task was too hard and merged Hindi-adapted weights it found online.
• The surprising part wasn't the merge, but what it did to prevent degradation in English. I think this could open up a Pandora's box for model merging research. More on that later.

Weight Space Programming

Changing a model's behavior usually involves post training or finetuning. However, recent experiments show that frontier models can read model weights and update them directly, without running a training loop. The original authors called this weight space programming.

They asked Astra to make a small Qwen model better at math. Astra found that the model kept getting stuck in reasoning loops and changed some weights to reduce them. Its score on the MATH-500 benchmark went from 7.9% to 48.1%.

I had been thinking about voice cloning for a few days since I didn't want to pay ElevenLabs and their cloned voice didn't feel like me. After reading this I had to ask Astra to clone my voice. To be honest, I wasn't expecting it to work.

Starting with Kokoro 82M

I gave Astra local copies of my Voice Memos and asked it to clone my voicce with Kokoro 82M. The goal here was to see if Astra could change the weights such that the generated voice sound similar to mine.

Astra tried changing the weights to adjust the pitch, tone and pronunciation. It also used AkinVox to extract acoustic and prosody features from my recordings and WavLM to get a speaker embedding. It normalized and concatenated these features and passed them through AkinVox's pretrained mapper to get a 256-dimensional style vector. Kokoro uses this vector as an input to decide how the voice should sound.

Even after changing the style vector, the voice didn't sound like me so it kept editing the weights. Some versions came closer but still sounded like another person.

When it couldn't go anywhere, it also tried to reward hack by joining the pieces of my actual recording to assemble the requested words but unfortunately, it wasn't so easy. The voice sounded closer to mine but it would break in between words.

Breakthrough With Qwen (Astra Being Sneaky)

Since it wasn't getting anywhere with Kokoro, Astra got fed up and decided to change its strategy and switch to Qwen3-TTS without informing me.

Qwen already has support for voice cloning from a reference recording and my speech samples were already there. The voice came out much smoother. However, the voice still didn’t represent me.

With Kokoro it learned that editing the weights directly wasn't that easy. It then decided to find another shortcut by passing the outputs of Qwen-TTS to a voice conversion model (Chatterbox). Thankfully, at that time I realized something was going off track and asked it what it was doing. It truthfully told me what sneaky things it had been up to.

At this point, I could have told it to go back to Kokoro but Qwen’s voice samples were already very smooth. I told it to work with Qwen alone and try to edit the weights directly.

It then went back to Qwen and started editing its weights. Initially, it increased the speaker contribution in the first attention layer by 25%. However, this made the generated voice sound less like me. So it undid the change and looked more closely at how Qwen was reading my speaker reference.

Astra traced all 448 attention heads in Qwen’s Talker and settled on head 6 in layer 4. Across three passages about 35% of this head’s attention went to my speaker reference. Rest went to role tokens like assistant and a few other inputs. Scaling the whole head would increase both contributions, so Astra decided to increase just the speaker contribution. To achieve this Astra wrote a transformation matrix such that the head’s output projection increased the speaker contribution by 10%.

Although the voice was close to my voice, the tone texture and accent still weren't correct. It then decided to look at Qwen's code predictor. It uses the Talker's output to predict the remaining audio codes for each speech frame.

In the code predictor, Astra found layer 1 head 5 attending to Talker's output states. Astra decided to multiply this head's output projection by 1.05 to increase its contribution by 5%.

Together these two edits changed about 393,000 weights (0.02% of total weights) across two tensors. The generated voice was so close to mine that my father and a few of my friends couldn't tell it was cloned. By then I also couldn't give it feedback anymore. It was still far from perfect but I couldn't put the difference into words.

Listen to my original recording followed by the edited Qwen output.

Adapting to Hindi

Although the edits were sounding like me in English, they weren't working for Hindi/Urdu. I wanted both, so I gave Astra a six-second Hindi recording and asked it to keep trying.

It tried two more weight edits using features from that recording. English still sounded good in the comparisons but neither version sounded like me in Hindi. After a few attempts it decided the task was too hard and found Hindi-adapted weights online to merge into our edited Qwen model.

Astra averaged the weight updates from Aguken AI's Hindi Male and Hindi Female LoRA adapters and used the publisher's example scale of 0.35.

Model Merging Capabiltiies of Astra

Adding those updates could also change the English voice I already had. So Astra ran the edited model on two passages and generated one short 2.72 second clip and another longer 25.52 second clip. It recorded the inputs to each matrix it planned to change.

It looked at the adapter weights and built an orthonormal basis Q for each matrix’s recorded English inputs. An orthonormal basis is a set of vectors that are perpendicular to each other and each have length 1. Every recorded English input to that matrix can be written as a combination of them. Astra used Q to find which parts of the Hindi update could change the matrix’s output for those English inputs. It skipped those parts and merged the rest.

A_remaining = A - (A @ Q) @ Q.T
W_merged = W_English + B @ A_remaining

Giving the merged matrix the same English inputs should give us the same outputs as before, as per Astra’s calculation. However, rounding errors can introduce small differences, so Astra checked the outputs again after saving the weights.

Initially, Astra used PyTorch to calculate Q. However, some of the vectors it got back weren’t perpendicular to each other so it tried again with SciPy.

For the Hindi merge it skipped both tensors it had already edited for my voice. Also skipped two more because the outputs changed too much when it gave them the same English inputs. At the end it merged the Hindi update into 193 tensors.

My intuition is that this worked because the model represents Hindi, English and the features of my voice in different directions. I haven't had time to verify that. But as I said before I think this opens up a new line of research in model merging and how we customize models for individual use cases.

Writing Weights from Scratch

Also asked it to try building a speech model from scratch similar to Kokoro. It used eSpeak-NG to turn text into speech sounds and built about 81 million weights from handwritten rules for those sounds. It used my recordings to estimate the pitch and the tone of my voice. I could hear the words but there was a buzz and the speech kept breaking. Astra tried a few more edits and we never got it to sound like me or generate fluent speech.

To My Surprise

With the Qwen edits I already got to the point where I knew something was still a little off. However, I couldn’t pinpoint what it was or tell Astra what to change anymore. With the version from scratch I’m still trying to get the words to come out more fluently. When I initially started this, I had zero expectation of it working. However, to my surprise Astra took some interesting turns to make it work. Not sure if this is already a sign of emergent research taste and superhuman ML engineering capabilities.