By Roman Aglaganov
Block’s new speech model, Dessa, is remarkable in its demo recordings. It laughs at the right moment, gasps at a surprise, says “yeah” while you are still talking. Block says it runs on a high-end MacBook. The company is releasing it carefully, first to safety researchers. Others will not be so careful, and voices this real will soon be easy to come by.
The cost of a convincing voice
For years the clumsiness of machine voices was itself a safeguard. A slight delay, a flat tone, a turn-taking pause: people could tell. That safeguard is going. Phone scams already use cloned voices of relatives. An AI that can hold a warm, natural, real-time conversation, on a laptop, with no cloud provider to monitor it, makes those scams cheaper and easier to run at scale.
Block argues that hearing emotion makes voice AI safer, because an assistant that can tell when someone is distressed can respond better. That is true. But the same skill that lets a machine hear distress lets it fake warmth, and fake warmth is how many frauds work.
One simple rule
Detection will not keep up; it never has with deepfakes. Disclosure is easier. Any AI voice talking to a member of the public should say, at the start, that it is an AI, and say so again if asked. Legitimate businesses lose almost nothing from this: a bank’s voice assistant works just as well when the customer knows it is one. Fraudsters will ignore the rule, of course, but a clear rule makes the fraud easier to prosecute and gives people a simple test: if a voice will not say whether it is human, hang up.
The EU’s AI Act already requires AI systems that interact with people to be designed so that people are told they are dealing with an AI, unless it is obvious. Other countries, and the companies building these models, should adopt the same principle now, while realistic voices are new, rather than after the first wave of scams.
Build it in
Model makers can help directly. A voice model that introduces itself as an AI by default, and that is trained to answer honestly when asked, would make disclosure the normal behaviour rather than a setting someone has to switch on. Block has asked researchers to find the risks it has not thought of. This one is already obvious.
