Build your own voice agent

How to Build an Inbound AI Voice Agent From Scratch (Without Vapi)

By Shreyas Raj. Written from my YouTube video, about 51 minutes long.

Video: I Reverse-Engineered Vapi (And It's x10 Cheaper) on the AI with Shreyas Raj YouTube channel.

Short answer: You can build an inbound AI voice agent that books appointments by wiring together a phone number provider, LiveKit for audio, Sarvam for speech, an LLM, Supabase and a calendar tool, then hosting it on a VPS. In the video I walk through the full setup and the cost logic behind it.

What you will learn

Why a self-built stack can cost far less per minute than hosted voice platforms
The tools in the stack and what each one does
How to collect API keys and fill in the environment file
How to connect a phone number to LiveKit with a trunk and a dispatch rule
How to deploy the agent and dashboard on a VPS with Coolify

Why build your own instead of using a platform

Most people who start an AI voice agency go to hosted platforms like Vapi, Retell or Bland. In my experience they are expensive once you move into production. Underneath, they run a fairly simple backend on LiveKit with agent orchestration, and you pay for the convenience of simple login, testing and deployment.

So I spent about a week building the whole system myself. In the video I quote a cost of roughly 10 rupees for a five minute call, which works out to about 2 rupees a minute, and I compare that with the 9 to 15 cents a minute that hosted platforms commonly charge. Treat those figures as my own estimate for my setup, not a promise of what you will pay.

In the video I say you may need upwards of 1,000 to 1,500 rupees to follow along: the VPS runs about 500 to 800 rupees a month and the phone number about 600 to 800 rupees a month. You can find free VPS options, but plan for that minimum.

The dashboard I show is similar to what hosted platforms give you: a calendar of bookings, total calls, average duration, a CRM list of callers, call logs with downloadable transcripts, API credential settings, model and voice selection, an editable opening line and system prompt, and a listening sensitivity setting for noisy environments.

The stack and what each piece does

Before touching any code, here is the stack I used and the job of each part. A voice agent needs a way for people to reach it, a way to hear, a brain, a voice, a place to store data and a place to run.

Vobiz: the telephony provider where you buy the phone number
LiveKit: the real time audio pipeline that orchestrates speech to text and text to speech
Sarvam: speech to text and text to speech, handling Hindi and English
OpenAI GPT-4o mini: the brain (you can swap in Groq or another provider)
Supabase: stores call logs, transcripts and recordings
Coolify on a Hostinger VPS: runs the agent around the clock
Telegram bot: sends a notification when a booking is made
A calendar tool such as Cal.com: makes the actual booking

Step 1: Buy a number and connect it to LiveKit

Sign up on Vobiz, top up your balance and buy a number from the numbers section. Then go to SIP trunks and create an inbound trunk. To get the URI for that trunk, open your LiveKit project, go to telephony, SIP trunks and copy the SIP URI. Add it in Vobiz as a new URI (you do not need the sip colon prefix), save it, and select it as the primary URI for your trunk.

Next, link your purchased number to that trunk. Then go back to LiveKit and create a new inbound trunk there with the same number. The number now routes calls from Vobiz into LiveKit.

Step 2: Get the code and fill in the environment file

Download the code repository as a zip, unzip it and open the folder in an AI code editor. I used Google Antigravity, but any editor works. Rename the example environment file to the real environment file. This is where all your keys live.

You will need keys from LiveKit, OpenAI, Sarvam and Cal. The ones people usually get stuck on are Supabase and Telegram.

Supabase: copy your project URL and your key, create a storage bucket named call recordings, then copy the S3 endpoint, access key ID and secret access key
Telegram: message BotFather, create a new bot, copy the token, send your bot a message, then read the chat ID from the bot updates
Cal: create an event type, copy the event ID, then create an API key under developer settings
LiveKit: create a key in settings and copy the environment variables it gives you

Step 3: Run the agent locally and create a dispatch rule

Create a virtual environment, activate it, and install the requirements file with pip install -r requirements.txt. Then start the agent in dev mode. When it is running you should see your agent appear in the agents section of LiveKit.

Copy the agent name from there, go to dispatch rules, create a new rule, enter the agent name and select your trunk. Now call your number. If you get errors, paste them into your AI editor and ask it to fix them. In my case the issue was simply that I had not saved the environment file.

Once a call goes through, check your calendar for the booking and your Telegram for the confirmation message.

Step 4: Set up the dashboard and database

Start the UI server locally to see the dashboard, which reads from Supabase. For a fresh project, open the SQL editor in Supabase and run the SQL from the setup file once. Create the call recordings bucket first, then run the SQL, which is the order I recommend.

The dashboard lets you change the opening greeting and system prompt from the UI, and those changes reach the backend.

Step 5: Deploy on a VPS with Coolify

Push your code to a new GitHub repository (your AI editor can do this if you give it the commands GitHub shows). Then get a KVM VPS on Hostinger, choose the India region for the best latency, and pick the Coolify panel option. In the video I use the KVM 2 plan.

In Coolify, create a project, add a resource from a private repository with a GitHub app, and load your repository. Two settings matter a lot: the branch should be main and the build pack should be Dockerfile. Expose port 8000, paste your environment variables, and deploy.

Stop the local agent before testing, otherwise calls do not know whether to go to your machine or the server. Then place a call and watch the logs.

Where to go from here

The video also covers ways to cut cost further, such as self hosting LiveKit, and quality upgrades like ElevenLabs or Deepgram. In the video I give latency as roughly 800 to 1,500 milliseconds. If you would rather not build this yourself, that is the kind of work my team at RapidXAI does for businesses.

Watch the full 51 minute walkthrough to see every click, every error and every fix as I build and deploy this voice agent live.

Watch the full video See EducateXAI

Results vary and depend on your effort. Read the results disclaimer.

Questions people ask

Do I need to know how to code to build this?

Not deeply. I use an AI code editor and paste errors into it so it fixes them. You do need to be comfortable copying commands, creating accounts, and setting up API keys and environment variables carefully.

What does the setup cost to run?

In the video I say plan for at least 1,000 to 1,500 rupees to start: a VPS at roughly 500 to 800 rupees a month and a phone number at roughly 600 to 800 rupees a month, plus usage fees.

Why choose the India region for the VPS?

Latency affects how natural a voice agent feels. I mention that choosing a US region pushed latency up to around 383 milliseconds, which hurts performance, so I pick India for callers in India.

What are the most common deployment mistakes?

In Coolify the build pack must be Dockerfile and the branch main, with the correct port exposed. Also remember to save your environment file and to stop the local agent before testing the deployed one.

More from the channel

Prefer short reads? Start with the EducateXAI guides.