The Dawn of AI Support: Our First Experiments with Lyro (June '23) - Part 1 of 3

Before & After

Before Lyro

• We had 200–300 chats every day.
• NPS: ~9.0
• CSAT: 4.4
• Teams had to be big to manage the volume of inquiries, which caused management overhead.
• People were overwhelmed and close to burning out by replying to the same basic questions over and over again.

After Lyro

• We have 200–300 chats a month at most.
• NPS ~9.0 (no change)
• CSAT: 4.4 (no change)
• We saw good results from day 1 and never stopped improving the resolution rate—now at ~70%.
• Lyro sticks to its safety guardrails, and thanks to using Claude by Anthropic; it’s unlikely to hallucinate.
• Implementation was based on scraping our website, historical conversations, articles from the help center, and a bunch of Q&As written on the spot. Lyro’s knowledge base quickly grew to over 2000 questions and answers in our support project.

Our journey to AI-first customer support began in June 2023, and this is the first in a three-part series where we’ll share our transformation story.

As one of the very first teams to test Tidio’s own AI agent, Lyro, live on our homepage, we were eager—and nervous—to see what it could do.

Those First Steps: Promising, Imperfect, and Exciting

We ran our initial test on a small batch of conversations with real users. The results? Close to a 40% resolution rate. Honestly, for a brand-new tool in its earliest phase, this felt extremely promising. It was already a huge improvement over the static FAQ buttons we had before.

More importantly, that first test showed us exactly where to focus.

We saw that about half the time Lyro couldn’t answer, it was because our knowledge base content was lacking, or we didn’t consider that the same question could be asked differently. The other half involved tweaking Lyro’s “fallback” responses to “no-context” messages like a simple “thanks” or “sure”.

But these were minor hiccups. What mattered most was how Lyro dealt with facts—

Only one hallucination in that entire batch of hundreds of messages. This was an astounding result.

🙌 Learning, Improving, and Implementing

Those early learnings were invaluable. As it turned out, the main challenge wasn’t the AI technology itself—it was getting our input information right. We needed to pump out more content, especially around AI topics, and get it organized and “digestible” for Lyro.

Even with that initial learning curve, the actual technical setup later on was incredibly fast—quicker than onboarding a new human agent. We gathered our help center articles, internal docs, and Q&As, uploaded them, and Lyro quickly started connecting the dots.

💡 From “Just” a 40% Resolution Rate to Being AI-First

Seeing those early, promising results (even at a “mere” 40% RR) fueled our excitement. After refining our content and Lyro’s settings based on those initial tests, we saw resolution rates climb higher and faster, eventually reaching 60–70%, even with those early models.

From that promising start in June 2023 to becoming a fully AI-first support team, it’s been quite a ride. We’re incredibly proud to use our own cutting-edge tech to serve our customers better. Could we imagine going back? No way. Lyro is essential now.

Stay tuned for part 2 of our AI support journey, in which we focus on the impact on the team. Coming soon.