Adding a Japanese Voice Chat AI to My Portfolio with VOICEVOX × Next.js

I added a voice chat feature to my portfolio site (meetyudai.com): you talk into the microphone, and it answers back in a natural Japanese voice. The speech synthesis is powered by VOICEVOX, a free, high-quality Japanese TTS project.

A talking portfolio is memorable, and there are surprisingly few practical write-ups about running VOICEVOX in the cloud — so here's the architecture, the implementation, the pitfalls, and the actual costs.

What is VOICEVOX? (30 seconds for the unfamiliar)

VOICEVOX is a free Japanese speech synthesis project with a roster of character voices ("Zundamon," "Shikoku Metan," and others) whose quality rivals commercial TTS. The crucial part for developers: besides the GUI app, the synthesis engine ships as an official Docker image exposing a plain HTTP API — so you can run it server-side and call it like any REST service.

• Voices are selected by a speaker (style) ID — e.g., Zundamon (normal) = 3, Shikoku Metan (normal) = 2. GET /speakers lists them all
• Character voices require attribution (e.g., VOICEVOX:Zundamon); terms vary per character, so check the official site

Architecture overview

Four players: the browser, Firebase Functions (the API), OpenAI (speech recognition + response generation), and the VOICEVOX engine on Cloud Run.

┌──────────┐  ① POST MP3 recording ┌─────────────────┐
│ Browser   │ ────────────────────▶ │ Firebase Functions │
│ (Next.js) │                       │  /voice-chat       │
└──────────┘                       └────────┬─────────┘
     ▲                                      │
     │                       ② Transcribe with Whisper
     │                       ③ Generate reply with ChatGPT
     │                                      │
     │                                      ▼
     │                             ┌─────────────────┐
     │  ⑤ Return WAV (base64)      │ VOICEVOX Engine  │
     └──────────────────────────── │ (Cloud Run)       │
                                   │ ④ audio_query     │
                                   │    → synthesis    │
                                   └─────────────────┘

In words:

1. The browser records mic input as MP3 and sends it to the API
2. Firebase Functions transcribes it to Japanese text via the Whisper API
3. The text goes to ChatGPT, which generates a reply
4. The reply goes to the VOICEVOX engine, which synthesizes a WAV
5. The WAV returns base64-encoded and plays in an <audio> element

Why not use a cloud TTS for everything? OpenAI's TTS could return audio too. But for a Japanese voice with character — one visitors remember and smile at — VOICEVOX is in a league of its own, and that personality was the whole point for a portfolio.

Try it locally first (docker run + curl)

Before putting anything in the cloud, get a feel for the engine on your own machine — everything downstream gets easier. The engine starts with a single Docker command:

# Run the CPU build locally (port 50021)
docker run --rm -p 50021:50021 voicevox/voicevox_engine:cpu-ubuntu20.04-latest

Once it's up, open http://localhost:50021/docs in a browser. A Swagger UI (OpenAPI docs) ships inside the engine, so you can exercise every endpoint from the browser. This is the fastest way to learn the API.

Driving the two-step API by hand with curl looks like this:

# ① audio_query: text → synthesis parameters (JSON)
curl -s -X POST \
  "localhost:50021/audio_query?text=こんにちは、ようこそ&speaker=3" \
  > query.json

# Peek inside — morae, pitch accents, speed, all visible as JSON
# jq '.speedScale, .accent_phrases[0]' query.json

# ② synthesis: synthesis parameters → WAV
curl -s -X POST "localhost:50021/synthesis?speaker=3" \
  -H "Content-Type: application/json" \
  -d @query.json \
  > hello.wav

# Play it (afplay on macOS, aplay on Linux, etc.)
afplay hello.wav

Within seconds, Zundamon's voice comes out of your laptop. Doing this first builds intuition for where the pipeline is slow and what can fail.

Three tips for local experimentation:

• Japanese text pasted directly into a URL can break depending on your shell's encoding. If it misbehaves, use the Swagger UI at /docs, or route the text through the equivalent of encodeURIComponent from a scripting language
• Hand-edit query.json — set speedScale to 2.0, raise pitchScale — and you'll feel the design insight that the intermediate JSON is the tuning surface
• During development, point your local Next.js/Functions environment straight at http://localhost:50021 and the whole dev loop costs zero cloud money

Frontend: recording and calling the API

I used mic-recorder-to-mp3 for recording. With the raw MediaRecorder API you end up fighting per-browser output formats (Chrome gives you webm, Safari mp4…); a library that guarantees "you always get MP3" simplifies handing the file to Whisper.

'use client';
import { useState } from 'react';
import MicRecorder from 'mic-recorder-to-mp3';

const recorder = new MicRecorder({
  bitRate: 128,
  encoder: 'mp3',
  numberOfChannels: 1, // mono is plenty — half the file size
  sampleRate: 44100,
});

const VoiceChat = () => {
  const [isRecording, setIsRecording] = useState(false);
  const [botAudioURL, setBotAudioURL] = useState('');

  const startRecording = () =>
    recorder.start().then(() => setIsRecording(true));

  const stopRecording = () => {
    recorder.stop().getMp3().then(async ([buffer, blob]) => {
      const file = new File(buffer, 'input.mp3', { type: blob.type });
      const formData = new FormData();
      formData.append('audio', file, 'input.mp3');

      const response = await fetch(`${API_BASE_URL}/voice-chat`, {
        method: 'POST',
        body: formData,
      });

      // Turn the base64 WAV into a Blob URL and play it
      const base64Audio = await response.text();
      const audioBlob = await fetch(
        `data:audio/wav;base64,${base64Audio}`
      ).then((res) => res.blob());
      setBotAudioURL(URL.createObjectURL(audioBlob));
      setIsRecording(false);
    });
  };

  return (
    <div>
      <button onClick={isRecording ? stopRecording : startRecording}>
        {isRecording ? 'Stop' : 'Talk to me'}
      </button>
      {botAudioURL && <audio src={botAudioURL} controls autoPlay />}
    </div>
  );
};

Small gotchas:

• Mic permission only works over HTTPS (localhost is exempted, so dev is fine)
• iOS Safari is strict about autoplay — if autoPlay doesn't fire, call audio.play() from a user-gesture handler
• Release Blob URLs with URL.revokeObjectURL() when done to be kind to memory

Backend: Whisper → ChatGPT → VOICEVOX

The Firebase Functions side is a pipeline: receive audio, turn it into text, think of a reply, turn the reply into a voice.

export const voiceChat = onRequest({ cors: true }, async (request, response) => {
  const audioFile = (request as any).file; // multipart handled by multer

  // ① Transcribe with Whisper
  const transcript = await openai.audio.transcriptions.create({
    file: fs.createReadStream(audioFile.path),
    model: 'whisper-1',
    language: 'ja', // pinning the language improves both accuracy and speed
  });

  // ② Generate the reply with ChatGPT
  const chat = await openai.chat.completions.create({
    model: 'gpt-4o-mini',
    messages: [
      {
        role: 'system',
        content:
          "You are the guide of Yudai's portfolio site. " +
          'Answer in Japanese, warmly, in 2–3 short sentences. ' + // ← crucial: long replies = slow TTS
          'Briefly introduce his background and projects when asked.',
      },
      { role: 'user', content: transcript.text },
    ],
  });
  const replyText = chat.choices[0].message.content ?? '';

  // ③ Synthesize with VOICEVOX (two-step API)
  const wav = await synthesizeWithVoicevox(replyText, 3); // 3 = Zundamon

  // ④ Return as base64
  response.set('Content-Type', 'audio/wav').status(200).send(wav.toString('base64'));
});

That "answer in 2–3 short sentences" instruction is not cosmetic. Reply length translates directly into synthesis time and playback delay — in voice UIs, making the bot concise is latency engineering.

VOICEVOX's two-step API, and tuning the voice

The two-step API is the part that trips people up at first:

async function synthesizeWithVoicevox(text: string, speaker: number): Promise<Buffer> {
  // Step 1: text → synthesis parameters (JSON)
  const queryRes = await fetch(
    `${VOICEVOX_URL}/audio_query?text=${encodeURIComponent(text)}&speaker=${speaker}`,
    { method: 'POST' },
  );
  const audioQuery = await queryRes.json();

  // Edit the JSON here to tune the voice
  audioQuery.speedScale = 1.1;       // slightly faster, conversational tempo
  audioQuery.pitchScale = 0.0;       // default pitch
  audioQuery.intonationScale = 1.2;  // a bit more expressive
  audioQuery.prePhonemeLength = 0.1; // trim leading silence

  // Step 2: synthesis parameters → WAV
  const synthesisRes = await fetch(`${VOICEVOX_URL}/synthesis?speaker=${speaker}`, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify(audioQuery),
  });
  return Buffer.from(await synthesisRes.arrayBuffer());
}

/audio_query converts text into a JSON of synthesis parameters (readings, pitch accents, speed); /synthesis turns that JSON into an actual WAV. It's genuinely good design: the JSON in between is your tuning knob for speed, intonation, and silence — and if a word is mispronounced, you can fix the kana/accent phrases in the JSON before synthesis.

Choosing a speaker (character) and the /speakers endpoint

Which voice you pick matters more than it looks — the first impression of a voice chat is mostly the character of the voice.

GET /speakers lists everything available. The JSON looks like this (excerpted and simplified):

[
  {
    "name": "四国めたん",
    "speaker_uuid": "7ffcb7ce-....",
    "styles": [
      { "name": "ノーマル", "id": 2 },
      { "name": "あまあま", "id": 0 },
      { "name": "ツンツン", "id": 6 },
      { "name": "セクシー", "id": 4 }
    ]
  },
  {
    "name": "ずんだもん",
    "speaker_uuid": "388f246b-....",
    "styles": [
      { "name": "ノーマル", "id": 3 },
      { "name": "あまあま", "id": 1 },
      { "name": "ささやき", "id": 22 }
    ]
  }
]

The structure to internalize: it's a two-level hierarchy of speaker (character) → styles. The speaker parameter you pass to the API is actually a style ID, not a character ID. The same Zundamon sounds completely different in "normal" (3) versus "whisper" (22).

Practical selection tips:

• Match the voice to the site's tone. For a portfolio guide, a friendly "normal" style; for a calmer site, a lower voice. Since swapping styles changes the whole persona, synthesize a few candidates through /docs and listen side by side first
• Don't hardcode the style ID — push it to config. "Actually, let's change the voice" always happens eventually. Keep speaker in an environment variable or a config doc and you can swap voices without a deploy
• The /speakers result is safe to cache. It only changes when you upgrade the engine, so there's no need to fetch it per request
• Attribution is per character. If you use Zundamon, put VOICEVOX:ずんだもん somewhere on the site. Terms differ per character, so make it a habit to check the official terms page the moment you pick a voice

Running the VOICEVOX engine on Cloud Run

The engine ships as an official Docker image, so I pushed it to GCP's Artifact Registry and deployed on Cloud Run.

# Pull the official image and push it to Artifact Registry
docker pull voicevox/voicevox_engine:cpu-ubuntu20.04-latest
docker tag voicevox/voicevox_engine:cpu-ubuntu20.04-latest \
  us-west1-docker.pkg.dev/MY_PROJECT/voicevox/engine:latest
docker push us-west1-docker.pkg.dev/MY_PROJECT/voicevox/engine:latest

# Deploy to Cloud Run (CPU build, generous memory)
gcloud run deploy voicevox-engine \
  --image us-west1-docker.pkg.dev/MY_PROJECT/voicevox/engine:latest \
  --memory 2Gi --cpu 2 \
  --port 50021 \
  --no-allow-unauthenticated \
  --region us-west1

<!-- TODO(Yudai): confirm the actual region/specs/auth setup from the original deploy. -->

Do not leave the endpoint public

The VOICEVOX engine has no authentication of its own. Deploy with --allow-unauthenticated and anyone who finds the URL can synthesize speech on your bill. Deploy private and call it from Functions with an ID token:

import { GoogleAuth } from 'google-auth-library';

const auth = new GoogleAuth();

async function callVoicevox(path: string, init: RequestInit) {
  // Fetches an ID token with the Cloud Run URL as audience
  const client = await auth.getIdTokenClient(VOICEVOX_URL);
  const headers = await client.getRequestHeaders();
  return fetch(`${VOICEVOX_URL}${path}`, {
    ...init,
    headers: { ...init.headers, ...headers },
  });
}

Grant the Functions service account roles/run.invoker and you get service-to-service auth with no key files to manage.

The cold start problem

The engine image is huge (multiple GB); with min-instances=0 the first request can take tens of seconds. Three options:

For a "it only needs to work when I'm showing it" use case, a Cloud Scheduler ping every 15 minutes during the day — or a manual warmup before an interview — is enough. For an always-on experience, pay for min-instances=1.

Error handling and timeouts: "no voice" must not mean "broken"

This pipeline chains three external dependencies in series (Whisper, ChatGPT, VOICEVOX), which means any single one failing takes down the whole flow. And the most failure-prone of the three is VOICEVOX, with its cold-start baggage. Do nothing about this, and a merely sleeping engine turns your site into "press the button and it goes silent for a minute" — looking maximally broken at exactly the moment you most want to show it off.

There's one design principle: voice is the enhancement; text is the baseline. It's progressive enhancement, verbatim — if the voice can't be produced, at least answer in text.

First, always set timeouts. fetch waits forever by default, so cap it with AbortSignal.timeout:

async function synthesizeSafe(
  text: string,
  speaker: number,
): Promise<Buffer | null> {
  try {
    // A ceiling that tolerates a warm-ish cold start.
    // Past this, we accept "no voice this time."
    return await synthesizeWithVoicevox(text, speaker, {
      signal: AbortSignal.timeout(8000),
    });
  } catch (err) {
    // Timeouts and connection errors both land here.
    // A synthesis failure is NOT a conversation failure — don't rethrow.
    console.warn('voicevox synthesis failed, falling back to text', err);
    return null;
  }
}

Then change the API's response shape from "the WAV itself" to "text, plus audio if available":

// The response always contains text. Audio is a bonus.
interface VoiceChatResponse {
  replyText: string;          // the ChatGPT reply — always returned
  audioBase64: string | null; // null when VOICEVOX failed
}

const wav = await synthesizeSafe(replyText, SPEAKER_ID);
response.status(200).json({
  replyText,
  audioBase64: wav ? wav.toString('base64') : null,
});

The frontend plays the audio when audioBase64 is present, and renders the text in a speech bubble when it's null. To the user, that reads as "it answered in text today," not "it's broken" — an enormous difference in perceived quality.

Sorting out how each stage's failure should be handled:

One more practical note: set the Functions-level timeout longer than the sum of the inner ones. If you cap synthesis at 8 seconds internally but the function itself is force-killed at 10, the fallback path gets murdered before it runs. The rule: outer timeout > sum of inner timeouts + processing time.

Monitoring and logging: be able to say which stage is slow

With three APIs called in series, "it feels slow" is guaranteed to happen. If you've planted logs that can immediately answer which stage was slow, investigations shrink by an order of magnitude.

The recipe is simple: time each stage and emit one structured log line per request:

export const voiceChat = onRequest({ cors: true }, async (request, response) => {
  const timings: Record<string, number> = {};
  const stopwatch = (label: string) => {
    const start = Date.now();
    return () => (timings[label] = Date.now() - start);
  };

  let done = stopwatch('whisper');
  const transcript = await transcribe(request);
  done();

  done = stopwatch('chat');
  const replyText = await generateReply(transcript.text);
  done();

  done = stopwatch('voicevox');
  const wav = await synthesizeSafe(replyText, SPEAKER_ID);
  done();

  // One request = one structured line. Filter and aggregate directly in Cloud Logging.
  console.log(JSON.stringify({
    event: 'voice_chat_completed',
    timings,                            // { whisper: 900, chat: 1400, voicevox: 1100 }
    totalMs: Object.values(timings).reduce((a, b) => a + b, 0),
    transcriptLength: transcript.text.length, // log the LENGTH, never the content
    replyLength: replyText.length,
    audioSynthesized: wav !== null,     // feeds the fallback-rate monitor
    speaker: SPEAKER_ID,
  }));
  // ...
});

Log design notes:

• Never log the utterance or transcript content. You have no idea what a visitor will say into the mic. For debugging, what you need is almost always the length, not the content — so record transcriptLength only. It also makes the privacy posture of your logs much lighter
• Watch the rate of audioSynthesized: false. When it climbs, that's a cold-start or outage signal on the VOICEVOX side. Thanks to the text fallback, users won't report it — the log is your only tripwire
• Look at per-stage latency as a distribution, not an average. "Normally 3 seconds total, but 40 on a cold start" is bimodal; the average lands on a number that describes neither mode and hides the problem. In Cloud Logging, a log-based metric viewed at p50/p95 is the low-effort answer
• Send one line of client-side perceived time too, and the picture is complete: the gap between the server's total and the browser's "button press to first sound" is your transfer + decode + playback-start overhead

The interesting part is that what needs monitoring differs from a normal web app. Every HTTP status can be 200 while "30% of replies had no audio" — a half-broken experience. The key is to explicitly log business-level success (did the voice come out?), not just transport-level success.

Latency and cost in practice

The pipeline calls three APIs in series, so the experience is "speak, wait 2–5 seconds, get an answer."

stop recording → Whisper (~1s) → ChatGPT (~1–2s) → VOICEVOX (~1s) → transfer & playback

Rough cost per conversation:

• Whisper: $0.006/min → ~$0.001 for a 10-second utterance
• gpt-4o-mini: well under $0.001 for a short reply
• VOICEVOX: the software is free; the real cost is Cloud Run CPU time
• Total: well under a cent per conversation — noise at portfolio traffic levels

To make it feel truly conversational, the next step is sentence-level streaming: receive the LLM reply as a stream, cut it at sentence boundaries, send each sentence to VOICEVOX immediately, and play the WAVs in order. Perceived latency shrinks to "time until the first sentence is ready."

// Sketch of sentence-level streaming synthesis
let sentenceBuffer = '';
for await (const chunk of chatStream) {
  sentenceBuffer += chunk.choices[0]?.delta?.content ?? '';
  const match = sentenceBuffer.match(/(.+?[。!?])/);
  if (match) {
    enqueueSynthesis(match[1]); // ship this sentence to VOICEVOX now
    sentenceBuffer = sentenceBuffer.slice(match[1].length);
  }
}

Security recap checklist

Here's the checklist I walked through before going live. The trap in this architecture is that behind the "free engine" hang two metered, pay-per-use services (OpenAI and Cloud Run) — so what you're protecting is mostly your wallet.

• Is the VOICEVOX Cloud Run service deployed --no-allow-unauthenticated? The engine itself has no auth; the moment it's public, anyone can synthesize on your dime
• Is roles/run.invoker restricted to the Functions service account only? Check IAM for a stray allUsers or an over-broad role
• Is there a rate limit on the /voice-chat endpoint? Every hit on this endpoint incurs OpenAI charges. Even a naive per-IP "N requests per minute" cap is enough to stop scripted hammering
• Are upload size and recording duration capped? Limit recordings client-side (say, 30 seconds max) and validate file size server-side; oversized files hit both your Whisper bill and your memory
• Is CORS restricted to your own domain? Keep cors: true (allow-all) for development only; in production allow just https://meetyudai.com
• Are you sure no utterance content lands in the logs? Per the previous section: lengths and metadata only
• Are billing alerts configured? A GCP budget alert plus a monthly usage limit on the OpenAI side. When something goes wrong, these two are the last line of defense
• Is the character-voice attribution on the site? Licensing rather than security — but it belongs on the same pre-launch list

Every item takes minutes, but the rate limit and the billing alerts in particular are the insurance that keeps an accident from becoming an incident.

Takeaways

• The VOICEVOX engine runs beautifully as official Docker image → Artifact Registry → Cloud Run, exposed as a plain REST API
• Start with docker run + curl + the bundled /docs Swagger UI locally; the intuition you build makes everything after faster
• It's a two-step API (audio_query → synthesis); the JSON in between is your voice-tuning knob
• /speakers lists the voices; the ID you pass is a style ID, not a character ID — keep it in config so swapping voices needs no deploy
• The engine has no auth — keep Cloud Run private and use ID-token service-to-service calls
• Cold starts are real; choose between paying for min-instances and a warmup routine
• Fall back to a text reply when VOICEVOX fails — "no voice" must never read as "broken"
• Log per-stage latency and the "no-audio reply rate" as structured lines; never log utterance content
• Voice latency engineering starts with making the bot concise; sentence-level streaming is the next win
• Walk the security checklist before launch — what you're guarding is mostly your wallet
• Don't forget the character voice attribution

Most portfolios are things you look at. One you can talk to is remembered. Huge thanks to the VOICEVOX project for making Japanese TTS this accessible — next up: sentence-level streaming synthesis.