alt.hn

4/24/2025 at 5:10:14 PM

Show HN: Lemon Slice Live – Have a video call with a transformer model

by lcolucci

4/25/2025 at 8:45:27 AM

Just talked with Max Headroom and Michael Scott - my wife is an office fan so knows the references, and I know enough Max to ask the right things.

Overall, a fun experience. I think that MH was better than Scott. Max was missing the glitches and moving background but I'd imagine both of those are technically challenging to achieve.

Michael Scott's mouth seemed a bit wrong - I was thinking Michael J Fox but my wife then corrected that with Jason Bateman - which is much more like it. He knew Office references alright, but wasn't quite Steve Carell enough.

The default while it was listening could do with some work, I think - that was the least convincing bit; for Max he would have just glitched or even been completely still I would think. Michael Scott seemed too synthetic at this point.

Don't get me wrong, this was pretty clever and I enjoyed it, just trying to say what I found lacking without trying to sound like I could do better (which I couldn't!).

by djaychela

4/25/2025 at 3:33:20 PM

Thanks for the feedback. This is definitely a demo where every piece matters for maximizing the enjoyment factor. We spent the most effort on optimizing video quality and latency, but not a lot on tweaking the character prompts that go into the LLM. Turns out that matters a lot too.

by andrew-w

4/24/2025 at 11:51:37 PM

This is impressive. The video chat works well. It is just a hair away from a very comfortable conversation. I'm excited to see where you have it a year from now, if it turns out to be financially viable. Good luck!

by zebomon

4/25/2025 at 12:47:36 AM

Thank you! Very much agree that we need to improve speed to make the conversation more comfortable. Our target is <2sec latency (as measured by time to first byte). The other building blocks of the stack (like interruption handling, etc) will get better in the coming months as well. In 1 year things should feel like the equivalent of a zoom conversation with another human.

by lcolucci

4/24/2025 at 6:24:41 PM

I am very much fascinated by this virtual avatar talking thing. I tried video-retalking https://github.com/OpenTalker/video-retalking just to see how far I can make it work to make a talking avatar but it is tremendously difficult. But this holds tremendous possibilities and I hope it can be eventually cheaper to run such models. I know this is far superior and probably a lot different but I hope to find open source solutions like Lemon Slice someday that I can experiment with.

by srameshc

4/24/2025 at 6:30:44 PM

Nice! Thanks for sharing. I hadn't seen that paper before. Looks like they take in a real-world video and then re-generate the mouth to get to lip synch. In our solution, we take in an image and then generate the entire video.

I am sure they will have open source solutions for fully-generated real-time video within the next year. We also plan to provide an API for our solution at some point.

by sid-the-kid

4/25/2025 at 2:01:31 PM

So basically the old open-source live-portrait hooked up with audio output. Was very glitchy and low res on my side. btw: Wondering if it's legal to use characters you don't have rights to. (how do you justify possible IP infringement)

by mentalgear

4/25/2025 at 2:26:08 PM

One way this differs is in the model architecture. Our approach relies on a single pass of a diffusion transformer (DiT), whereas Live Portrait relies on intermediate representations and multiple distinct modules. Getting a DiT to be real-time was a big part of our work. Quoting the Live Portrait paper: "Diffusion-based portrait animation methods [...] are usually [too] computationally expensive." As you hinted at, we had to compromise on resolution to get there (this demo is 256x256), but we think that will improve over time.

by andrew-w

4/25/2025 at 2:33:00 PM

Not relying on facial keypoints means we can animate a wide range of non-humanoid characters. My favorite is talking to the Doge meme.

by andrew-w

4/25/2025 at 2:19:03 PM

> Wondering if it's legal to use characters you don't have rights to. (how do you justify possible IP infringement)

IP law tends to be "richer party wins". There's going to be a bunch of huge fights over this, as both individual artists and content megacorps are furious about this copyright infringment, but OpenAI and friends will get the "we're a hundred-billion-dollar company, we can buy our own legislation" treatment.

e.g. https://www.theguardian.com/technology/2024/dec/17/uk-propos... a straightforward nationalisation of all UK IP so that it can be instantly given away for free to US megacorps.

by pjc50

4/24/2025 at 5:58:45 PM

This is very impressive. Any details about model architecture and size? Input and output representation?

How does voice work? You mentioned Deepgram. Does it mean you do Speech-to-Text-to-Speech?

by lostmsu

4/24/2025 at 6:25:01 PM

For the input, we pass the model: 1) embedded audio and 2) a single image (encoded with a causal VAE). The model outputs the final RGB video directly.

The key technical unlock was getting the model to generate a video faster than real-time. This allows us to stream video directly to the user. We do this by recursively generating the video, always using the last few frames of the previous output to condition the next output. We have some tricks to make sure the video stays relatively stable even with recursion.

by sid-the-kid

4/24/2025 at 7:06:08 PM

I'm not at that level but reminded me of https://news.ycombinator.com/item?id=43736193

by tough

4/24/2025 at 7:17:01 PM

Nice find! I hand't seen this before (and will take a deeper look later). It looks like this is an approach to better utilize the GPU memory. And, we would probably benefit from this to get more of a speed-up, which would also help us get better video quality.

I do not think they are running in real time though. From the website: "Personal RTX 4090 generates at speed 2.5 seconds/frame (unoptimized) or 1.5 seconds/frame (teacache)." That means it would take them 37.5s to generate 1 second of video, which is fast for video but way slower than real time.

by sid-the-kid

4/24/2025 at 7:41:03 PM

Yep, this is way slower but considered SOTA on video-gen open source.

I mostly meant the using the previous frames to generate new frames insight that reminded me but lack knowledge on the specifics of the work

glad if its useful for your work/research to check out the paper

edit: the real-time-ness of it also has to have into equation what HW are you running your model on, obviously easier to make so on a H100 than a 3090, but these memory optimizations really help to make these models usable at all for local stuff, which is a great win i think for overall adoption/further stuff being build upon them a bit like sd-webui from automatic1111 alongside stable diffusion weights models being open sourced was a boom on image gen a couple years back

by tough

4/24/2025 at 10:06:58 PM

Nice. This is also how recent advances in ML weather forecasting work. Weather forecasting really is just "video generation" but in higher dimensions.

by tony_cannistra

4/24/2025 at 9:41:45 PM

Nice! What infra do you use for inference? I'm wondering what the cost-effective platforms are for projects like this. GPUs on AWS and Azure are incredibly expensive for personal use.

by dheera

4/24/2025 at 9:54:56 PM

We use modal (https://modal.com/). They give us GPUs on-demand, which is critical for us so we are only paying for what we are using. Pricing is about $2/hr per GPU (as a baseline of the costs). Long story short, things get VERY expensive quickly.

by sid-the-kid

4/24/2025 at 6:04:49 PM

thank you! We have an architecture diagram and some more details in the tech report here: https://lemonslice.com/live/technical-report

And yes, exactly. In between each character interaction we need to do speech-to-text, LLM, text-to-speech, and then our video model. All of it happens in a continuously streaming pipeline.

by lcolucci

4/25/2025 at 11:49:23 AM

This is fantastic. I was the founder of the 3D Avatar Store, a company that was doing similar things 15 years ago with 3D reconstructions of people. Your platform is what I was trying to build back then, but at the time nobody thought such tech was possible, or they seriously wanted to make porn, and we refused. I'll try reaching out through channels to connect with your team. I come from a feature film VFX, Academy Award quality work, so it would be interesting to discuss. Plus, I've not been idle since the 3D Avatar Store, not at all...

by bsenftner

4/25/2025 at 3:45:15 PM

We've been very inspired by interactive character experiences powered by traditional VFX + puppetry (turtle talk with crush is a favorite). I think that sort of interactive entertainment will become more commonplace as tech like ours continues to improve. Looking forward to connecting!

by andrew-w

4/26/2025 at 7:56:50 AM

this was overall fun. better than expected. i'm an office fan so tried dwight and michael scott. i hope you folks get better at this. excited to see where you get in the next 12 months or so. Godspeed!

by anishsikka

4/24/2025 at 7:21:19 PM

honestly this feels kinda huge - stuff like this is moving so fast, it's insane seeing it go real-time

by gitroom

4/24/2025 at 7:45:33 PM

IMO, most videos models will be fully real time within 2 years. You will be able to pick a model, imagine any world and then be fully immersed in it. Walk around any city interacting with people, first person shooter games on any map with crazy monsters, or just let the model auto-pilot an adventure for you.

by sid-the-kid

4/25/2025 at 9:26:07 AM

Probably not, but even if so, how much will that cost? There's AI that will take a pronpt like "what's the best weed whacker in 2025" and build a whole web page to publish the review. It's great, awesome. $10 in tokens to do that.

And that's probably still a subsidized cost!

Bfw "what is the best weed whacker" is John C. Dvorak's "AI test"

by genewitch

4/24/2025 at 7:44:18 PM

thanks so much for the kind words! we agree that the leap to real-time feels huge. so excited to share this with you all

by lcolucci

4/24/2025 at 9:05:01 PM

Hmm, plug this together with a app which collects photos and chats with a deceased love one and you have a working Malachim. Might be worth a shot.

Impressive technology - impressive demo! Sadly, the conversation seems to be a little bit overplayed. Might be worth plugging ChatGPT or some better LLM in the logic section.,

by elternal_love

4/24/2025 at 9:15:01 PM

Thanks for the feedback. Optimizing for speed meant we had fewer LLMs to choose from. OpenAI had surprisingly high variance in latency, which made it unusable for this demo. I think we could probably do a better job with prompting for some of the characters.

by andrew-w

4/25/2025 at 9:28:22 AM

You know the trick of having Gemini or mistral re-jigger the prompt?

Also, you do realize that this will be used to defraud people of money and/or property, right?

All about coulda, not shoulda.

by genewitch

4/25/2025 at 10:10:36 AM

> Also, you do realize that this will be used to defraud people of money and/or property, right?

Sadly, no one cares. LLM driven fraud is already happening and it is about to become more profitable.

by netdevphoenix

4/25/2025 at 8:18:43 AM

Hey, this looks really cool! I'm wondering - what happens if you feed it something totally different like a Van Gogh painting or anime character? Have you tested any non-photo inputs?

by NoScopeNinja

4/25/2025 at 3:06:57 AM

This is just brilliant. Hope you succeed, so that eventually I get an API to play with.

by ashishact

4/25/2025 at 3:50:55 PM

Thanks! What kind of use case are you thinking about?

by andrew-w

4/25/2025 at 5:52:09 AM

Really cool stuff. It felt strangely real. Impressive!

by wouterjanl

4/25/2025 at 3:53:39 PM

Thanks! We think we can cut down the latency to <2s which should make it feel even more natural.

by andrew-w

4/25/2025 at 2:25:09 PM

Are you going to offer a web embeddable version of the Live offering?

by o_____________o

4/25/2025 at 3:49:12 PM

It's something we are considering. What use cases do you have in mind?

by andrew-w

4/25/2025 at 5:37:30 PM

Mostly personal/art projects for now, but willing to pay.

by o_____________o

4/25/2025 at 12:41:02 PM

love the demo video with Andrew. showing the potential as well as the delays and awkwardness of AI is refreshing compared to the heavily edited hype reels that are so common

by inhumantsar

4/25/2025 at 2:07:04 PM

I spent about 2 hours recording videos with different characters. Of course, the one I made as a joke for myself and never intended to share was the most enjoyable to watch :)

by andrew-w

4/24/2025 at 7:38:45 PM

Very nice. Are you planning a paper?

by benob

4/24/2025 at 7:48:12 PM

thank you! No concrete paper plan yet as we're focused on shipping product features. anything specific you'd want to read about?

by lcolucci

4/25/2025 at 5:03:30 PM

watching baron harkonnen verbally create me code for todo list in React was rather amusing, thanks

by movedx01

4/25/2025 at 5:22:29 PM

glad to bring a little joy into the world :)

by andrew-w

4/24/2025 at 6:35:38 PM

The system just crashed. Sorry! Working on getting things live again as fast as we can!

by sid-the-kid

4/24/2025 at 6:42:20 PM

We are live again folks! Sorry about that. We ran out of storage space.

by sid-the-kid

4/24/2025 at 6:38:05 PM

Ah the ole HN soak test.

by PUSH_AX

4/24/2025 at 6:47:42 PM

Ya. You always think you cross your Ts. But, the law always holds.

by sid-the-kid

4/24/2025 at 7:45:52 PM

haha one of the reasons launching on HN is great!

by lcolucci

4/24/2025 at 9:05:41 PM

Max Headroom lives !

by aorloff

4/24/2025 at 9:40:55 PM

Just added as a public character :)

by andrew-w

4/24/2025 at 11:56:03 PM

Really wish his trademark glitching head nod was there, but I can imagine how that might not be possible.

Super cool product in any case.

by consumer451

4/26/2025 at 12:24:07 AM

Really well done

by aorloff

4/24/2025 at 9:23:48 PM

Does he? I can't find him.

by sid-the-kid

4/26/2025 at 12:24:27 AM

Toggle the tab to public

by aorloff

4/24/2025 at 9:31:03 PM

Looked it up. Cool reference.

by sid-the-kid

4/24/2025 at 5:12:45 PM

> reducing delays and improving resolution (purpose-built ASICs will help)

How can you be sure? Investing in an ASIC seems like one of the most expensive and complicated solutions.

by bigyabai

4/24/2025 at 5:16:42 PM

We wouldn't build it ourselves, but there are several companies like Etched, Groq, and Cerebras working on purpose-built hardware for transformer models. Here's more: https://www.etched.com/announcing-etched

by lcolucci

4/24/2025 at 11:58:27 PM

A really compelling experience.

It seems clumsy to use copyrighted characters in your demos.

Seems to me this will be a standard way to interact with LLMs and even companies - like a receptionist/customer service/salesperson.

Obviously games could use this.

by andrewstuart

4/24/2025 at 5:27:07 PM

If you could lower the email signup for a few hours, that'd be nice. I'm not going to sign up for yet another service I'm unsure about.

by tetris11

4/24/2025 at 5:34:49 PM

We just removed email signup. You can try it out now without logging in. It was easier than expected to do technically, so we just shipped a quick update.

by sid-the-kid

4/24/2025 at 6:30:12 PM

Thanks! This is amazing

by tetris11

4/24/2025 at 6:46:18 PM

Glad you like it! IMO, biggest things to improve on are 1) time to video response and 2) being able to generate more complicated videos (2 people talking to each other, a person walking + talking, scene cuts while talking).

by sid-the-kid

4/24/2025 at 5:24:38 PM

"Try it now live" and then request me to enter my email.

I'll pass thanks.

by doublerabbit

4/24/2025 at 5:33:51 PM

That's fair. We just removed the sign-in for HN. Should be live shortly.

Each person gets a dedicated GPU, so we were worried about costs before. But, let' s just go for it.

by sid-the-kid

4/24/2025 at 6:36:57 PM

I think it's not going well? I keep getting to the start a new call page, it fails, and takes me back to the live page. I assume your servers are on fire, but implementing some messaging would help ("come back later") or even better, a queueing system ("you're N in line") would help a lot.

Really looking forward to trying this out!

by sgrove

4/24/2025 at 6:46:22 PM

We're back online! One of our cache systems ran out of memory. Oops. Agree on improved messaging.

by andrew-w

4/24/2025 at 5:43:31 PM

Do you use a cloud-based GPU provider?

by yahoozoo

4/24/2025 at 5:48:21 PM

Yes. We use Modal (https://modal.com/), and are big fans of them. They are very ergonomic for development, and allow us to request GPU instances on demand. Currently, we are running our real-time model on A100s.

by sid-the-kid

4/24/2025 at 6:07:33 PM

I see you are paying $2/h. Shoot me an email at victor ta borg.games if your model would fit on RTX 3090 24G to get it down to $0.2/h (fellow startup).

by lostmsu

4/24/2025 at 7:16:03 PM

maybe demos could be a downsampled bitrate/size running on commercial GPU's

by tough

4/24/2025 at 6:21:20 PM

How much would this demo cost you from the HN traffic if you don't mind me asking?

by ivape

4/24/2025 at 6:44:54 PM

Good question. I guess depends on how many users we get. Each users gets their own dedicated GPU. Most video generations systems (and AI systems in general) can share GPUs during generation. Since we are real time, we don't do that. So, each user minute is a GPU minute. This is the biggest driver of the cost.

by sid-the-kid

4/24/2025 at 7:15:04 PM

feels like the next logical step for you to bring enconomies of scale is to allow users generating the video to automatically stream it to n platforms, so each gpu can be generating 1 png for many humans to watch simultaneously, with maybe 1 human driving the seat on what to generate, or more ai, idk

by tough

4/24/2025 at 9:37:47 PM

that's a good idea! Would be especially cool if the human is charismatic and does a good job driving the convo. Maybe we can try it out with a streamer.

by sid-the-kid

4/24/2025 at 9:59:48 PM

Vtuber comes to mind

by tough

4/25/2025 at 2:21:34 PM

Neuro/vedal already there, although not with the model as well.

by pjc50