Launch HN: Agnost AI (YC S26) – Extract user feedback from agent conversations

7/14/2026 at 11:54:34 PM

I see a fair number of comments here advocating for either codex to hand-roll this themselves, or to simply punt to SQL. I do want to advocate for the difficulty of the problem, even if I can't speak to the company itself.

At the scale of a few hundred to a few thousand documents, especially short documents, there are a few out of the box methods that can yield reasonable results, whether it be embedding clustering or leveraging LLMs for tagging.

However as your (1) datasets gets larger (2) documents expand from tweets and text messages to 30+ minute conversations and (3) you build downstream analytics on top of the learned semantic units, you really start to feel the limitations of LLMs and embedding for reliable annotation. That doesn't even get into the nuances associated with taxonomy management, seasonality, and model drift.

TLDR; this problem solved effectively has a lot of value and is a lot harder than it seems.

by kianN

7/15/2026 at 8:03:08 AM

Why is it hard? Ultimately you take whatever your signal is and send it to some relatively cheap LLM.

How is it easier to sign up and manage a different service, implement a different API, etc.

And from the company side the fatal flaw is that these types of tools rely upon 1% of their users having huge spend. Nobody is going to be a huge spender here because it's easier to hand roll than navigate procurement on this (not to mention impossible to justify the spend, additional security/privacy risk, etc.)

It feels approximately impossible for this company to have large accounts.

by bluelightning2k

7/15/2026 at 8:15:29 AM

it gets hard when you need this continuously across lots of chats/calls, with metadata, changing clusters, going deeper into a user journey, etc. the LLM call is just one part of it lol

we're keeping it useful every week, finding out insights that the teams can extract value out of, work with them to understand users better.

the procurement what we've seen is v similar to how one would have for any analytics product? and we're selling this to companies when/once it becomes someone's job to do this

by laalshaitaan

7/15/2026 at 11:22:41 AM

If you get this to work once rerunning it weekly seems fairly easy unless you actually need to see the data live and have perfect uptime?

by pqtyw

7/15/2026 at 5:55:30 PM

a lot of our customers want a daily morning report on slack & flag things instantly rather than to wait for a week so thats why we keep it realtime

by laalshaitaan

7/15/2026 at 1:19:11 AM

yea, at our volume which we still consider small as we've been able to figure out a way with llms & embeddings, its still fine. + we onboarded a voice ai company with more than 2 hour calls and thats when it was super hard to solve since there were so many elements to consider.

model drifting is something a lot of folks do face after 5th/6th turn as per my understanding and it usually the median, how did you tackle it if you have yet?

also yea, thats why we went for a per customer taxonomy than a general one, yeilded better results + easier to improve upon.

by laalshaitaan

7/15/2026 at 2:16:13 AM

To clarify, I wasn't criticizing your approach or product, more responding to the people dismissing the problem you are solving.

Regarding my experience, I have done a fair amount of work in the contact center space with long calls. I used statistical Bayesian approaches which I found to be much more resilient especially on long documents than embeddings/transformers. It also provided a joint modeling foundation for classification with much lower label requirements than BERT or traditional ML.

by kianN

7/15/2026 at 4:17:40 AM

im hearing this for the first time and damn! i just told this to my cofounder/cto and he said hes gonna give this a shot in the coming days.

damn, i read bayesian in statistics like years ago, never thought itll come back this way

by laalshaitaan

7/15/2026 at 4:52:11 AM

Happy to chat more in depth if more details would be helpful. I think my contact info is accessible from my HN profile.

by kianN

7/15/2026 at 11:20:37 AM

IMHO this would make more sense if provided as part of a larger "platform" like Langfuse/Langsmith/etc. Otherwise you just end with a dozen SaaS products for highly specific use cases which might not scale that well.

Realistically do you also need to have this live with a fancy? i.e. a custom solution maybe even Jupiter notebooks initially might be sufficient. It's not like 100k messages is a very large dataset. It's not trivial to make a generic solution that fits every use cases (besides of basic customer chatbots) to get actual value for more agentic products.

by pqtyw

7/15/2026 at 8:36:37 PM

i get the push, most teams come to us after they've done/tired of the claude running analysis thing manually and want a pro-active thing.

we're also targeting conversation first use cases and for them this serves as their everything custoemrs. we obv do not sell the fancy part, idts that sells anymore lol. lot of our queries come from our slack app/mcp.

by laalshaitaan

7/15/2026 at 8:59:23 PM

[dead]

by pqtyw

7/14/2026 at 11:28:29 PM

I built an in-house version of this a couple of years ago for where I was working. My concern would be that by excluding observability, you might end up creating a really selective dataset, whose conclusions you're then asking companies to take seriously when allocating resources to different possible roadmaps.

My guess would be that agent logs would highlight obvious feature requests and bugs for smaller companies - like customers expecting an AI video editor product to be able to add subtitles to a video by itself.

For larger companies who deal with a higher volume of inbound customer support / agent requests, there will probably be big, noisy, already-known-by-the-team query clusters that make up big portions of the dataset - for example, "billing issue with my subscription". After those big clusters you'll likely have a really long tail of different queries, and - without deep observability - no real way to rank their importance. I also think you'd be unlikely to understand the root cause of the product issue in a complex developed product with lots of users solely from agent logs. Most product teams can't make good product decisions consistently, and they're working with a lot more data.

If coupled with staying out of evals (which, btw, I wouldn't find trust-building, if I were a potential customer of yours), I think that it might be difficult to provide genuine value in this space for larger orgs - without evals it's easily dismissed as just fancy & mostly-contextless sentiment analysis.

But I hope I'm wrong! I do think that (though each org's needs probably have to be catered to in a very boutique way) there are huge gains available by rolling LLMs & language analysis into existing product workflows, and that what you're pitching is absolutely a part of what companies should be doing. We are, of course, meant to actually listen to customers - and LLMs/agents should be making that easier, not harder. Absolute best of luck!

by gabriel666smith

7/15/2026 at 1:22:27 AM

i like these kinds of critiques, we don’t think conversation logs or analysis on top of it is alone enough to replace observability or evals. imo they answer diff questions for diff use-cases.

we're betting that there is a TONNN of product signal buried in conversations that observability misses, esp around like raging, writing in all caps, repeated prompts, frustration loops, and subtle hidden feature demand. thats also why we use per-customer taxonomies instead of a shared one. evals will still be needed.

the root cause is harder, especially in more mature agents. we're using this more as a discovery layer for evals or even just whats happening kind of things, then letting teams go deep into the actual conversations and decide what to take action upon

by laalshaitaan

7/15/2026 at 10:13:35 AM

There's definitely a tonne of signal in those, and it's a critique made from a place of strong support of your basic thesis. There's always been a tonne of signal in traditional customer support requests that goes un-used by most orgs, especially b2c orgs.

In case it's helpful: I always explained it to people I was training like this: All lean product theory comes from listening to the workers actually assembling the parts at Toyota.

Now, most digital products - whether the UI is graphical or linguistic - require a customer to work on an assembly line themselves. An onboarding flow is an assembly line and the user has tasks. Those users complain to agents (whether human or LLM) about their task on the assembly line. The purest implementation of lean philosophy would start with modelling these messages and conversations before it did anything else.

If I were you, I'd build a CRM. Intercom and its ilk charge ridiculous money for functionality that the people using it despise. The existing products in the space optimise for 'serve customers quickly' (increasingly irrelevant with LLMs) and not 'learning from your customers' (increasingly relevant as humans talk to customers less day-to-day). They are horrible to try to integrate into an established product development cycle (I've tried).

I think this makes the proposition easier to comprehend to a customer, the value-add more obvious, and allows you to undercut on pricing, rather than giving people a new bill for something they don't know if they need. The MVP of a CRM is also perhaps easier to build than it might seem initially. "Serve customers faster, cheaper, and learn from them in a highly configurable & meaningfully better way, giving your product iteration an advantage over your competitors". Building a CRM, crucially, allows you oversight of much more of the data - which then enables significantly more meaningful discovery.

This is the unsolved half of the coding agent space: what to actually build, what order to build it in, and why. It's really solvable from your starting point, and is potentially just as important/disruptive as the coding agent has been thus far - especially now that we suddenly have more lines of code than we know what to do with.

I'll shut up now - it's a fascinating space to me, so it's easy to get carried away about! Always happy to talk about stuff like this via email (in my profile) on the off-chance any of the above was useful, though :-)

by gabriel666smith

7/15/2026 at 6:08:17 PM

its fascinating how the toyota example comes up anywhere, its so good!

wdym by modelling the messages and conversations though? i lose you a bit there! for the crm approach, i do think it'll be a problem at some point right now.

the replacing budge is an interesting piece, we did not think of it yet, yes let me dm you on x!

by laalshaitaan

7/14/2026 at 5:29:41 PM

> Rageprompting

Lovely name! I implemented profanity monitoring in my Hermes setup to identify "learning opportunities" for my agents. It is quite useful. If you are budget-conscious, one challenge is determining what is the smallest number of previous rounds that Hermes needs to correctly infer what it did wrong. Curiously, Claude Code is horrible at figuring out what it did wrong. I often read its memories, and they are rarely useful.

by m_kos

7/14/2026 at 5:38:06 PM

haha yea, i even got the domain rageprompt dot com like a couple of days ago lol i love the name too.

for profanity, did you define keywords or just let the agent figure out rage stuff?

how many rounds did you set for the hermes? claude doesnt work yea on its own, one of my friends set us up for their claude lol

by laalshaitaan

7/14/2026 at 8:16:02 PM

Without using agnost, what are some basic SQL queries I can run on my data to find outliers I'd otherwise be missing?

How far can I get with just keywords, common phrases, boring traditional analysis?

Depending on what I measure there, when is the right time for me to consider upgrading to something like Agnost/what is a specific example of what it will find that traditional/rigid analytics approaches will miss?

by benswerd

7/14/2026 at 8:46:37 PM

keywords and sql rarely work - you can not find the repeated hidden feature requests, cause we don't know them at the first place yet, or a frustrated user puts vague signals as ugh, ahh, or just an 'f!' (and added modalities, accents and languages makes it much more challenging)

interestingly, even embeddings seem to bucket "no" and "nooo!" somewhat similar, but are pretty different when viewed from a user satisfaction perspective.

A sweet spot on moving to Agnost is the time when you get higher inflow of conversations you can't manually read or listen, and want to clusterize them into things which matter, with the outliers highlighted

by AjmeraParth

7/15/2026 at 12:31:01 AM

[flagged]

by StackOptimist

7/14/2026 at 9:37:55 PM

Well, good luck with the launch, this seems like an interesting product with potential.

However privacy is central in a service like this and I think you should probably beef up your representation of how you deal with that.

eg. "We use each customer’s data only for that customer" - well that customer may have hundreds of staff; how are they being consulted and onboarded wrt their own voices (or is that transcripts?) and messages being used in this way?

ofc you might argue that nothing in work is private but I do think you have some margin for improving the detail here.

by mellosouls

7/14/2026 at 10:10:49 PM

[dead]

by laalshaitaan

7/15/2026 at 1:40:48 AM

the hard part isn't extracting quotes, it's attribution – separating what the user actually felt from the agent's own framing, and sentiment that flips inside one session.

by rjnz199

7/15/2026 at 5:46:06 AM

did you actually use AI to write that? it sounds like an "the hard part is not A but B" which is literally what every LLM model generates

by vivzkestrel

7/15/2026 at 1:53:17 AM

seems ai slop

by laalshaitaan

7/15/2026 at 12:12:15 AM

Great launch!! There’s a lot of very silly comments of people saying they will vibe code this… errr good luck being the slop version of this startup. :/

It’s a cool product and I’m curious to see where you go. We build an MCP factory, where our enterprise customers use our product to build MCPs that their employees use in Claude or Codex. What would be cool for me is if I could use this to surface insights to them, rather than just to our team.

by r_thambapillai

7/15/2026 at 1:25:39 AM

i love this comment bec we started as analytics for mcp servers haha! we then expanded to conversations bec thats where most mcp servers were being used lol.

we havnt figured out yet how to do the b2b2b kinda thing where we surface insights for a multi-tenant sort of approach but i've gotten this now twice in the last hour so happy to chat.

what would you want them to see first though, are these semantic insights like we do today or more around deterministic tool calls/etc metrics?

by laalshaitaan

7/15/2026 at 1:32:52 AM

our current prototype of this functionality is fairly basic and surfaces things from the underlying chats like: Users want a "create ticket tool", or "The run SOQL query often fails because it references fields that don't exist. You should provide documentation of the actual fields". Happy to chat if you want to you can find me on bookface as the founder of credal

by r_thambapillai

7/15/2026 at 1:51:42 AM

oh gotcha, yeah happy to chat more! yeah DMing

by laalshaitaan

7/14/2026 at 10:36:08 PM

My junior developer has a Claude Cowork skill she built to do this over about 25,000 messages a week to our agent, and it seems to work pretty well. Struggling to understand what $499/month would buy us here?

by petesergeant

7/14/2026 at 10:41:12 PM

oh damn, can you share what the skill is actually doing for you like on a daily basis: is it creating clusters, scoring known issues, or finding new patterns? and what data points are you giving it to if any w the messages?

usually a boundary for us is usually where a skill/claude analysis needs to maintain/make changes/pass it to an agent as a workflow

for the pricing we're still learning and building as many custom features/requirements as possible bec we wanna make sure we deliver way more value than what we charge today.

by laalshaitaan

7/14/2026 at 10:51:42 PM

Classification of types of user frustrations and sentiment analysis, content trends, engagement and gap analysis, as well as then looking at changes from the previous week. We also look at how certain queries turn into actions in the system (eg: which users take actions we offer them). We run it once a week, rather than every day, and it provides an exec-facing overview, as well as areas for support to dig further in to. While it's some good work, as far as I'm aware it's almost all just a text prompt and a connection into Langfuse.

by petesergeant

7/14/2026 at 11:10:28 PM

okay nice, also is it safe to assume you do once a fortnight releases then? like look at the last week's data then use it for product decisions the coming week?

also have you updated/made any changes to this skill that has improved it significantly?

and anything you hate/wish it had as of today? wanna learn if there's any painpoints around this? is it keeping the skill updated, getting useful signal from the clusters, or turning the findings into something the rest of the team can actually act on?

by laalshaitaan

7/14/2026 at 5:25:50 PM

why would i pay $499/month for this when codex costs $199/month and can do everything you described

by zuzululu

7/14/2026 at 5:45:30 PM

codex is great for like a one-time/overview analysis on a handful of transcripts. we usually serve to companies where the volume is >10k messages & continuous ingestions + with claude/codex it messed up this + metadata linking of the user like what plan are they on, when is it expiring, etc.

although we had a few customers who come to us after running this for a while so at smaller volume it does work well.

by laalshaitaan

7/14/2026 at 6:25:09 PM

i mean i would get codex to build everything you just described

by zuzululu

7/14/2026 at 8:55:02 PM

Do it then.. the hubris of vibecoders is really something.

by dakolli

7/14/2026 at 10:37:10 PM

Reminds me of what people been spitting in my face (with a slight variation) for much of my career:

> A (vibe) programmer knows the value of everything, but the cost of nothing

by embedding-shape

7/15/2026 at 9:30:56 PM

To be fair if you have some background in data science (I think the main thing is being able to verify its not showing nonsense) it shouldn’t be too hard to build a bespoke solution.

by wasfgwp

7/14/2026 at 6:42:45 PM

Would you?

Looking forward to your "show HN" post.

by ImPostingOnHN

7/14/2026 at 6:30:59 PM

lol true but then you’re just building another us :D

by laalshaitaan

7/15/2026 at 7:56:31 AM

Why... why do companies keep taking every tiny feature and trying to productize it?

In the tradition of boring software, even before LLMs it was much simpler to just use your existing tools and hand-roll. With LLMs I cannot fathom reaching for a product for something small like this.

by bluelightning2k

7/15/2026 at 8:10:38 AM

i agree w you for smaller teams tbh. if you have a few hundred convos and someone can maintain scripts/prompts, hand-rolling is probably fine.

it becomes less tiny when it’s 10k+ msgs/week, long voice calls, metadata, changing clusters, retention/redaction, and the team wants this continuously without maintaining another internal tool.

but i'd still wanna know how you'd do this with existing tools before llms?

by laalshaitaan

7/14/2026 at 7:42:32 PM

I thought startups wrapping prompts would require something a more complex than semantic analysis, which is literally what this is. And for 500 bucks. Wow. Props for being able to sell this.

I don't get the appeal of the UI, why is it so complex/convoluted.

by lnenad

7/14/2026 at 8:26:13 PM

lol i wish it was just wrapping prompts but things got harder once our customers grew bigger, we had to build queues. we had to do context management for bigger conversations and bunch of metadata fields started coming in per customer.

by laalshaitaan

7/14/2026 at 8:33:23 PM

It's still a prompt, it's just not a static one. Either way props for building a company from it.

by lnenad

7/14/2026 at 9:23:50 PM

we're still learning and so our the prompts haha, whats your take though

by laalshaitaan

7/14/2026 at 8:57:03 PM

How is it just a prompt? Like hey, I hate AI companies with a passion but I think this is a lot more than just a prompt.

by dakolli

7/14/2026 at 8:58:24 PM

I don't hate AI companies. The key value proposition is gather data > feed it to AI for semantic analysis (does the actual work, is a prompt) > display it in a UI

by lnenad

7/14/2026 at 9:27:19 PM

on a satirical note: we also have an mcp server/api endpoint if you dont want the ui

by laalshaitaan

7/15/2026 at 4:07:28 PM

[flagged]

by CCsecurity3

7/15/2026 at 5:10:40 PM

[flagged]

by dnkzm

7/14/2026 at 11:29:05 PM

[flagged]

by synapsehire

7/15/2026 at 4:20:55 AM

[flagged]

by sahitya_

7/14/2026 at 4:27:48 PM

[flagged]

by WangYixiao

7/14/2026 at 9:44:55 PM

[flagged]

by czeizel