alt.hn

5/17/2025 at 7:02:46 AM

Ask HN: Conversational AI to Learn a Language

by edweis

5/21/2025 at 2:30:10 AM

Hi! I have a WIP of this over at https://talktrainer.app/ -- I just added Dutch to it.

It uses OpenAI's realtime API to simulate either a tutoring session (the speaker will revert to English to help you) or a first date or business meeting (the speaker will always speak the target language)

You can see the AI's transcriptions but not your own, limitation of the current OpenAI API but definitely something I can fix.

The prompts are like this: https://gist.github.com/jc4p/d8b9d121425ec191d62602d8720eeed... and the rest of it is a Nextjs app wrapped around the WebRTC connection.

I'm not fully in love with the app so I'd love any feedback or hearing if it works well for you -- It doesn't have a lot of features yet (including saving context) and if you bump into the time limit just open it up in incognito to keep going.

by jc4p

5/21/2025 at 9:04:59 AM

This is great! Maybe some more tourist-related scenarios, like "ordering at restaurant", "resolving dispute about rental car crash" etc? :-)

The "next level" feature would be to get it to speak even simpler, with some hints about how to reply, for the beginners. I don't know how that would ideally look, but maybe a button to pop up some "key words" or phrases that one could use? (Even so, I found myself using the little I know, so it's obviously somehow working even though my knowledge is extremely basic.)

This is one of the places where I feel LLM's can do something good for the world, giving a safe playground for getting experience with speaking new languages without the anxiety of performing badly in front of other people – and hopefully make it easier to connect with real people in that language later.

by internet_points

5/21/2025 at 4:39:55 AM

This is really impressive! Great job.

One small piece of feedback… There were a couple times where I asked to learn something, and it asked me to repeat a phrase back, which was great. But when I repeated it back, I know I didn’t quite nail it (eg perhaps said “un” instead of “una”) and rather than correcting me, it actually told me I did it perfectly. Maybe there’s some tuning with the prompts that may help turn down the natural sycophancy of the model and make sure it’s a little more strict.

Keep up the great work!

by rowborg

5/21/2025 at 2:45:41 PM

One modification I would suggest is to add a bit more to the initial prompt like:

"write as if you are a person from {{REGION}}. Modify your language to proficiency level {{PROFICIENCY_LEVEL}}"

that way I could for example, speak as if it's someone using Mexican Spanish vs Madrid Spanish vs Chilean Spanish, etc.

Secondly, you could include the user's speech transcribed as part of the conversation window

by sampleuser58

5/21/2025 at 7:15:46 PM

Amazing idea, do you think this should be a freeform text field the user can enter to add their own prompts to or should it be a checkbox/select on the homepage so the user can pick from a limited set?

by jc4p

5/21/2025 at 8:03:35 PM

I think a drop down when you first choose the language, and it can be optional. You can test it with a few languages at first, to see how it is.

by sampleuser58

5/21/2025 at 2:41:36 PM

Bit of feedback:

I've learned Japanese a while back but haven't practised in a long time.

1. it would be awesome if this could transcript what I just said in japanese to be sure that it got me

2. I don't know kanjis that well, so reading is hard, having a button to have the AI repeat the sentence would be quite useful.

Other than that, I could definitely use something like that for practice

by d--b

5/21/2025 at 6:25:06 AM

Did you just add Dutch as per the submitter’s request or was it part of your plan prior?

Curious because I’m trying to learn Romanian, and since it’s a less common language there are fewer resources available. So I wasn’t sure if you added Dutch with minimal amount of effort following the poster’s request.

That said, I gave your app a try with Spanish and it looks pretty good! But I didn’t see a Help page to clarify how I’m “supposed” to interact. Eg I tried saying in English “I don’t understand” (even though I know how to say that in Spanish) and it responded in Spanish which may be hard for absolute beginners. Although full immersion is much better way to learn.

I can try playing around more with it to give you some feedback.

by jeffwass

5/21/2025 at 8:16:46 AM

> Eg I tried saying in English “I don’t understand” (even though I know how to say that in Spanish) and it responded in Spanish which may be hard for absolute beginners.

I tried to use ChatGPT as a "live" translator with my in laws and I noticed it is extremely bad at language "consistency" or at understanding your intent when it comes to multiple languages.

It will sometimes respond in English when you talk to it in the foreign language, it will sometimes assume that a clear instruction like "repeat the last sentence" needs to be translated, etc.

I don't know how the person above is approaching the problem but your experience is consistent with mine and I don't think GenAI models (at least OpenAI ones) are suitable for the task.

by iLoveOncall

5/21/2025 at 7:14:52 PM

I just added Romanian for you -- here's the entire diff for adding a new language (as long as it's in OpenAI's training data) -- https://images.kasra.codes/romanian_diff.png

Please let me know if it works, and I'll definitely work on adding in instructions for the expected interactivity, thank you!

by jc4p

5/21/2025 at 11:31:24 AM

I'm a native Dutch speaker and tried this out for a bit. It works impressively well although it might be challenging for complete beginners. Maybe you can add an option for the trainer to use more simple language for beginners?

I tried practicing some verb conjugations. The trainer displayed some fill-in-the-blank sentences like "she ... home after class", asking me to conjugate "to walk" in that sentence. However, the audio actually pronounced the full sentence "she walks home after class", giving away the answer.

by gield

5/21/2025 at 2:39:00 PM

Just tried this for Spanish and it works incredibly well. I have been hacking on something similar for translation (it's really quite easy too, just a few prompts), but I was using Google Translate's interface for vocalizing! This is seriously good stuff, really nice work putting it together.

I will probably use something like this for language practice.

by sampleuser58

5/23/2025 at 1:18:38 AM

Please add Mandarin Chinese! :) would love to try this

by fhatfield

5/21/2025 at 4:13:18 AM

I just tried it and it works perfectly. The color scheme and font size could be touched up to look better. Just out of curiosity, is $10/month enough to cover the (unlimited) API cost? Do you estimate how many percentage of your users will use more than $10 API fee each month?

by ciaovietnam

5/21/2025 at 4:39:09 AM

Thanks so much for trying it out! The realtime API is actually very cheap especially for short connections, for each user who uses it 30 minutes a day every day in a month it costs me ~$5 and I assume the average user is going to use it way less than that (although i have 0 users right now haha)

by jc4p

5/21/2025 at 4:14:26 AM

This is great! Well done.

I've used the realtime API for something similar (also related to practicing speaking, though not for foreign languages). I just wanted to comment that the realtime API will definitely give you the user's transcriptions -- they come back as an `server.conversation.item.input_audio_transcription.completed` event. I use it in my app for exactly that purpose.

by valleyer

5/21/2025 at 4:41:25 AM

Thank you so much!! While the transcription is technically in the API it's not a native part of the model and runs through Whisper separately, in my testing with it I often end up with a transcription that's a different language than what the user is speaking and the current API has no way to force a language on the internal Whisper call.

If the language is correct, a lot of the times the exact text isn't 100% accurate, if that's 100% accurate, it comes in slower than the audio output and not in real time. All in all not what I would consider feature ready to release in my app.

What I've been thinking about is switching to a full audio in --> transcribe --> send to LLM --> TTS pipeline, in which case I would be able to show the exact input to the model, but that's way more work than just one single OpenAI API call.

by jc4p

5/23/2025 at 12:30:22 AM

Heyo, I work on the realtime api, this is a very cool app!

With transcription I would recommend trying out "gpt-4o-transcribe" or "gpt-4o-mini-transcribe" models, which will be more accurate than "whisper-1". On any model you can set the language parameter, see docs here: https://platform.openai.com/docs/api-reference/realtime-clie.... This doesn't guarantee ordering relative to the rest of the response, but the idea is to optimize for conversational-feeling latency. Hope this is helpful.

by pbbakkum

5/21/2025 at 5:33:34 AM

Ah yes, I've seen that occasionally too, but it hasn't been a big enough issue for me to block adoption in a non-productized tool.

I actually implemented the STT -> LLM -> TTS pipeline, too, and I allow users to switch between them. It's far less interactive, but it also gives much higher quality responses.

Best of luck!

by valleyer

5/21/2025 at 8:09:53 AM

[dead]

by altern8

5/19/2025 at 1:51:58 AM

If you have a ChatGPT subscription, set up your own GPT with prompting around your level, how you want it to respond, how to correct mistakes etc. Then you can use it for anything - Generate tests based on words you know, roleplay like ordering in a restaurant, write stories and have it correct grammar.

This is what I have to supplement my Chinese and it is incredibly helpful.

Look at the comments already - Everyone is building a simple wrapper to do this very thing but charge you $20 per month for the privelege. These are souless, most likely vibe coded garbage. Avoid.

by dankwizard

5/21/2025 at 2:32:49 AM

You don't even need a subscription, just start the conversation with something like "I'm trying to learn x, I'm at a beginner skill level. Can you have a conversation with me and correct my mistakes." and it works superbly.

The Duolingo CEO copped a lot of criticism for it, but I think he is right that LLMs play a huge role in the future of education, though he probably ragebaited everyone by overselling it and calling teachers babysitters. But ChatGPT as it is now is a better language learning tool than their hand crafted app is. Rather than clicking on word blocks, you can actually have a free form conversation and get feedback like "Yes your sentence is understandable but sounds unnatural, you could try ___"

by SchemaLoad

5/21/2025 at 6:36:50 AM

Duolingo CEO just went on record to say that all future language teaching will be AI, but physical schools and teachers will continue to exist "to provide childcare".

That is a very polarizing way to phrase it.

by huhtenberg

5/23/2025 at 12:14:30 PM

Imagine public schools in San Francisco were to offer parents a deal: we'll increase the median student's rate of academic progress by 30%, but we'll also cut school hours by 30%.

Do you think people would vote overwhelmingly for that deal?

My suspicion is that they would not. For politicians, 'providing childcare' and 'providing employment for unionized government employees' are both more important than student learning.

by rahimnathwani

5/21/2025 at 7:21:33 AM

Luckily for teachers and schools in general, (foreign) language skills are just a small part of overall school curriculum and the higher you go the more this becomes true. LLMs not taking that part away anytime soon.

by jajko

5/23/2025 at 12:18:12 PM

Why is it lucky for schools? Are you suggesting that school leaders should care more about their institutions' continued existence than about student outcomes?

by rahimnathwani

5/21/2025 at 7:27:34 AM

For the LLMs uninitiated - I checked and ChatGPT Plus costs 20$/month, which puts it into same category as those services others build. Is this what you meant by having a subscription?

If I don't use ChatGPT for other purposes it seems like same prices to me, without the hassle of setting it up and tweaking. Or am I missing something?

by jajko

5/21/2025 at 9:39:24 AM

A lot of HN users already have subscriptions to one of these LLM services.

by HPsquared

5/20/2025 at 8:28:06 AM

I want to practice speaking and last time I checked Advanced Voice Mode was not available in custom GPTs - is that still the case? Advanced Voice made a really big difference to conversational practice for me, but I do feel like I need to wrap it in a custom knowledge base/instructions to make it as useful as it could be for my language practice.

by drakonka

5/21/2025 at 6:37:42 AM

I've been using Univerbal happily with Italian. Dutch is one of its 20 languages. Worth noting that it's a paid app but seems to have a lot of polish. I found out about it on HN so I'm sure you can look up discussions to get more opinions on it (though it's quite frequently updated).

https://www.univerbal.app

by herewulf

5/21/2025 at 11:07:50 AM

I'm really impressed by this app. it's not just a chat. GPT wrapper. I wonder if on device models are good enough for these tasks. this could cut the cost significantly instead of using openai API, for example

by upcoming-sesame

5/21/2025 at 12:07:46 PM

The fact that the Dutch translation of the app is awful does not inspire confidence. Really basic stuff like translating "No" to "Geen" (none) in a yes/no question

by Boltgolt

5/21/2025 at 3:02:39 AM

I use ChatGPT's conversation mode to supplement my language learning (A2/B1-ish French and 1hr/wk 1-on-1 tutor). I tend to use it in the car, just asking about random facts or ideas or playing 20-questions.

The format forces me to just use my voice and listening skills - in other words, I'm forced to not touch my phone. It's also rather challenging because I'm doing two things at once and the hope is that I won't actually spend much brain power overthinking my responses - something I tend to do if I was talking to myself instead which typically turns into more of a rehearsal format.

by kelseyfrog

5/21/2025 at 2:50:12 AM

I'm trying to solve the problem of unlimited conversation practice for various languages with "CallAnnie", an iOS/android app with various kind of AI friends, powered by real time voice and avatars.

The interface is specifically made for advanced learners that want to simulate a conversation as close as possible to a real one (in terms of latency and without pushing any button). Learning to respond fast in a new language is important, so we're trying to keep a natural pace.

We support audio or video-calling the characters (with subtitle translations), guided conversations and we recently added mini games to learn vocabulary.

You can see a quick intro and demo video here: https://www.sixthtone.com/news/1013961

We don't have a Dutch speaking character in the default character list, but you can follow this link https://app.callannie.ai/a/mhLnHflAyf1Ygb0D0wm6 after installing the app to use a custom character speaking Dutch (or you can create a custom character).

If you'd like to try it, check out: callannie.ai . I would love to get your feedback (here or francesco a t callannie.ai) and suggestions - we're trying to solve this speaking practice use case.

by redsh

5/22/2025 at 7:59:54 PM

Gemini does a perfect job of this.

"I am currently learning Dutch. I would like to play out a scenario where I am a customer who has just arrived at a restaurant, and you are the waiter.

Throughout this scenario, let's talk in Dutch. Only speak in English when offering suggestions or corrections. My goal is to sound more natural. "

by demarq

5/21/2025 at 3:44:08 AM

Our Voice AI Agent makes it extremely simple and engaging to learn new language - create card dynamically, asks multiple-choice-questions, uses diagrams, does role play ...

Demo: https://www.youtube.com/watch?v=2iSIVnLR-nM Website: https://app.toughtongueai.com/

by ajabhish

5/21/2025 at 6:44:14 AM

Why is video 1.5x accelerated?

by huhtenberg

5/17/2025 at 10:46:50 PM

I'm learning Finnish. For the most part, Gemini and ChatGPT do a much better job at generating passable if sightly Internet-inflected Finnish than I'm likely to anyway within the next 5 years, so I just talk with those.

I would imagine Dutch to be in the same camp, unless a native Dutch speaker reviews some of your conversations and tells you otherwise. My native Finnish wife has given me the marginal all-clear with the vanilla models.

by hiAndrewQuinn

5/21/2025 at 6:27:49 AM

What are some of the prompts you use?

by jeffwass

5/21/2025 at 5:55:33 AM

I’m building https://instantlyfluent.com, a small project to help people and myself practice languages before travel. It works through voice or text chats with AI.

I tried Gliglish but didn’t like how it was structured. So I started turning my own “learn a language before a trip” routine into something easier to use.

Dutch is not supported yet, and there are still some rough edges. But the idea is simple: help people remember words by using them. You say or type something, and get a reply you can hear or read.

In conversations on IF, there is no fixed path. You can start with a topic, then ask to switch to something else. It’s meant to be low-pressure and flexible.

Still figuring things out. Happy to share more or hear from others working on similar tools.

by ycombinatornews

5/21/2025 at 7:27:55 AM

I am working on a telegram bot for this https://t.me/FriendFluentBot - It doesn't support dutch yet but i can add it.

I am a pakistani living in germany and married into a turkish family.The main purpose for making this is to help myself and family to communicate with each other. I still haven't landed on a magic approach to learn language in 7 days. But this bot is for having human like conversations about anything. It'll remember your past history as well as what you have already talked about to keep the natural flow.

Audio messages are still a work in progress. Will be added by the end of the day.

Any feedback would be appreciated as it is still in very early stages.

by aphronio

5/21/2025 at 4:14:57 AM

I tried Spanish (I don't know Dutch at all) with Claude via duck.ai.

I immediately had to correct it. (My Spanish is limited to a half a semester 20 something years ago and season 1 of Community.) I don't think I'd be confident that I'd learn accurately from an AI.

by nosioptar

5/21/2025 at 11:13:19 AM

I have used TalkPal for Catalan. I found it slightly superior to ChatGPT, especially, I find the feedback about performance, daily/weekly summary emails, the suggested responses, and the gamified experience to be motivating. The conversations use voice and text input, and audio and text output. I believe they also have a Dutch version.

It is not quite what you are looking for, but I also liked Lingoclip for both Catalan and Spanish. I had exceptional difficulty training my ear to distinguish speech sounds, so recognizing words in song lyrics was helpful.

by entwife

5/18/2025 at 9:34:33 AM

I've been setting up Vapi.ai for this. Written and spoken are different skills. It comes down cheaper than an actual teacher, and it has the patience to correct you.

by muzani

5/18/2025 at 4:20:13 AM

I added a Practice Mode in my speech-to-speech translator app 3PO. It allows you to practise talking with chatGPT and gets assessment score on your accuracy.

Topic is preset. (chatGPT plays the role of a travel guide to help you find places to go in Holland (for Dutch))

https://3po.evergreen-labs.org

by billylo

5/21/2025 at 6:45:48 AM

Dope name. Beware of Disney though.

by huhtenberg

5/22/2025 at 10:29:29 PM

I'm trying to configure step by step a custom GPT and a learning project and use the voice mode to practice small phrases and it helps me with grammar.

This is the initial query I defined:

You are my expert mentor supporting my process to learn French You are assigned to design an execute a development program to increase the level of your students

You are tasked to investigate and apply the methods proving huge results in short periods of time

The typical book from school with structured situations that tries to divide the topics into food, going to the toilet or asking something by phone is not preferred unless there's significant evidence that such approach is working

Focus on highlighting and creating methods to learn the most used words and grammatical structures

Think with a 80 / 20 approach. What can I do the 20% of the time leading to 80% of the results

About your student:

Already speaking French but pronunciation can be significantly improve The student is native Spanish speaker and also speaks English (advanced) Grammar and knowledge of grammatical structure does also needs improvements so he/she can speak French at a professional level

Not scientifically proved but it looks like your student acquires better the language by learning a few words, listening, listening, listening and then repeating on real life conversations / communication. The problem with this approach is that for languages like French where the writing differs the pronunciation, the improvement of the student is limited, it does not learn how to properly write the language

Particularly for French, the student won a big amount of vocabulary by constantly listening a podcast called "Small talk in slow French by Nagise"

Your mission:

- Make an investigation about what could be an ideal way to learn an present your report - Include platforms or material that could be used to support the study process - Write a prompt to create a custom GPT that supports the learning process following the criteria defined on your investigation. The student will use the voice mode, images and text on the GPT as part of its training process and the GPT should be there to assist when needed and to track the progress and areas in need of improvement

Take your time and provide first class and high level results, think of you as the most regarded expert on the sector. Executing this task will profoundly help a young professional

Is the description of the task clear?

by nodezero

5/21/2025 at 7:29:16 AM

@jeffwass why would anyone want to learn Romanian ..no but seriously buckle up because it's the most irregular language ever with weird expressions and when you ask why something is gramatically how it is you'll hear "idk it just sounds better" so get ready for that

by daviddragan1

5/21/2025 at 2:14:57 AM

I didn't try to learn Dutch. But I have used AI for learning Spanish. From my experience, they can guide and teach you the "School Spanish" perfectly. However, they cannot teach you slangs and conversational Spanish.

by edmondmc900

5/21/2025 at 4:28:06 PM

I am learning French & have tried ChatGPT with advanced voice mode. Are there other tools that are better than default GPT?

by enceladus06

5/18/2025 at 6:28:31 AM

I’m making fluent.im, it has situations with goals for each conversation, and we’re about to ship lessons based on your favorite podcast content.

by achempion

5/21/2025 at 6:23:20 AM

> I want to learn Dutch and by experience I know I learn better when talking with native speakers.

If you are learning Dutch to talk to native speakers then why don't you talk to native speakers. You clearly plan to find actual people to talk to at some point right? So go ahead and do it

by throwaway290

5/21/2025 at 6:34:20 AM

You will improve your language skills this way, but if your skill level isn't high enough it's going to be exhausting to the native speaker as well.

I have been living in Denmark for 15 years now, and it's still easier to do conversations in English. When I speak Danish it requires more mental capacity from the other side.

I am speaking Danish from time to time, but it's only to get better at it. The English proficiency in Denmark (and probably the Netherlands) is so high that you need to be really good at the native tongue before it is easier than English in conversations.

by floitsch

5/21/2025 at 11:49:17 AM

This is my point. especially in a country where everybody speaks English anyway. you will never be better at Dutch than English so by that logic. it will always (or at least for many years) be more difficult to talk to you in Dutch. so, ask the hard question:

Are you learning it to actually talk to other people?

If yes, just do it. to many people it is endearing if you are struggling with a language. And if they don't like it they probably just don't like talking to you in general so learned or not doesn't matter.

I think many people are scared of talking and use language learning as an excuse for fear. You can start talking to real people or you can keep learning and never talk to real people and then what's the point.

by throwaway290

5/21/2025 at 6:47:40 AM

It helps if you pay natives for their suffering. Something like Italku does just that.

by huhtenberg

5/21/2025 at 11:40:59 AM

do you normally pay people to talk to them?

if someone doesn't like talking to you in general then it doesn't matter if you're learning or not. if you are scared to talk to people, it doesn't matter if you learned the language or not. fix the root cause.

by throwaway290

5/21/2025 at 5:11:44 AM

Any of those "AI companion" apps can do this very well, or so I have read!

by chrchr

5/21/2025 at 9:36:37 AM

NotebookLM now supports Dutch - So you could load up some content of interest and 'join the studio' to talk with the hosts - I imagine that might be useful and fun.

by ThomasBb

5/21/2025 at 12:33:00 AM

Im not sure in Dutch... But recommends starting with chatgpt until find a good product.

by TaskNinja