A few years ago, clean transcription pasted at your cursor was novel. Today, there are hundreds of voice-to-text and cleanup models and many dictations products to choose from. For leaders in the voice dictation space, accuracy and latency improvements are both hitting ceilings.
We started working on the voice dictation anyway and deliberately. Industry-leading dictation that converts what you say into clean polished text pasted at your cursor. We shipped it and users found real value, helping them write several times faster than using a keyboard.
And it was technically challenging to pull off. Getting to 80% of the way there took only a few days. The last 20% of polish, bringing down error correction rates, handling various post-processing scenarios, getting the entire pipeline's latency to occur in milliseconds, that took us weeks to nail.
But voice dictation is just the stepping stone to a computer understanding your intent. The next leap to how people interact with their computers is a voice intent turned to action.
Voice to intent
Ever since I got my first iPhone in high school, Siri has been one of my favorite features. Speaking my intent and watching Siri make it happen felt very magical. Answering questions, checking the weather, setting up alarms, it's impressive what we were able to achieve before LLMs.
Today we have the capabilities to do more than basic voice commands. AI is incredibly accurate at deciphering a user's intent, and agents are getting increasingly good at doing real work. Instead of asking Siri to open up the Gmail app for you, what if you could ask the agent to summarize the most important emails that came in, help you draft responses.
That is a simple example, but it can go beyond that. It can write code for you, do creative work, visually generate UI and teach you new things. Having an assistant available to help you do work at all times. All you have to do is press a button and ask. This type of interface does not exist today.
That's what we're working on at Freestyle.
Upcoming 1: Freestyle Remix Writing Assistant
Today, Freestyle users use Freestyle dictation to write much faster than they can type. However, they're still copying what they write into AI for polishing, deep research, fact checking, content generation, then pasting it back into where they were typing.
Freestyle Remix writing assistant brings AI writing capabilities to your cursor, allowing you to make edits in place with full control. You interact with Remix by holding a hotkey and speaking to it. Remix will understand your intent and make the proper edits at your cursor.
Here are some example use cases:
- Highlight the citations that you've included in your essay, ask Freestyle Remix “Can you convert this citation into MLA format” and it'll rewrite your citation into MLA format.
- You're creating a powerpoint about the Great Depression. “Can you find that famous great depression image of the migrant woman?”. Remix finds the image and inserts it for you.
- You highlight a couple of rows on an excel sheet. “Can you turn this into a list that I can post on Slack?”. Freestyle gives you something shareable.
- Highlight a piece of text and ask “can you translate this into Mandarin and Spanish?”
Upcoming 2: Freestyle Producer Agent
Once we've achieved Remix as a writing assistant, we want to extend its capabilities to Producer, the agent that can do anything you can do on your computer. This is a system that will understand your intent and perform real work across all of the integrations that are hooked up to it.
Producer will have the capability to connect across many apps that you use, making anything searchable and bridging everything under a single interface you talk to. It can generate UI overlays to teach you how to use Photoshop, help you study a new chapter, and more.
Examples of what Freestyle Producer could do:
- You're learning Photoshop. "How do I mask out this background?" Producer draws an overlay on your actual screen, pointing at the actual tools in your actual UI. Not a tutorial you have to translate into your own workspace.
- Describe your least favorite type of work. "Go through my Downloads folder, rename the invoices by vendor and date, and file them into the right Drive folders." Freestyle Producer gets it done.
- “Where did that boarding pass go for my flight to New York”? Freestyle Producer searches across your email inboxes, computer storage, anywhere it has access to in search of your boarding pass.
Upcoming 3: Presets
Dictation apps today are good at general speech-to-text but still don't perform well where it matters for your job. For software engineers, "kernel" comes back as "colonel." For healthcare workers, "palpation" and "palpitation" get swapped, two very different words.
Presets are pre-configured pipelines that specialize in one kind of work. Choose the software engineer preset and Freestyle pre-loads a vocabulary and post-processing prompt built for engineering: it writes markdown, respects snake_case and camelCase, handles variable names, and can write code for you.
For healthcare, we load up a preset dictionary of medicines and disease names and even route the transcription request to models that are trained in medicine.
Help us build it
Freestyle is an open source project. We're a small community of developers working on voice dictation and intent to action. If you're interested in working on stuff like this, please consider checking out the GitHub repo and joining our Discord channel. All skill levels welcome!
