Cross-Dialect Text-To-Speech in Pitch-Accent Language
Incorporating Multi-Dialect Phoneme-Level BERT

Kazuki Yamauchi, Yuki Saito, Hiroshi Saruwatari
The University of Tokyo, Japan.

1. Samples of synthetic speech

We present speech samples synthesised using the following models:

●FS2: The original FastSpeech2[1]
●FS2-AP (proposed): The proposed model using ALVs predicted by the ALV predictor
●FS2-AP-Scratch: FS2-AP without initialing the model parameter on the basis of the phoneme-level BERT[2] (PL-BERT) pre-training
●FS2-AP-W/O: FS2-AP incorporating PL-BERT pre-trained without data augmentation by LLM-based dialect translation
●FS2-REF (proposed): The proposed model using ALVs extracted from reference speech
●FS2-REF-F0: FS2-REF using fundamental frequency (F0) as prosody feature instead of bottleneck (BN) feature

Speech samples synthesised by inputting texts sampled from transcriptions in CPJD-Osaka[3] into each model are shown in the table below. The target speakers for intra-dialect TTS (ID-TTS) and cross-dialect TTS (CD-TTS) were defined as for the JMD[4] Osaka-dialect speaker and the JSUT[5] Japanese standard speaker, respectively. Note that the target dialect for the ALV predictor is Osaka-dialect. Reference speech sampled from the CPJD-Osaka are also shown (REF in the table).

ID-TTS: Synthesis of Osaka-dialect speech by Osaka-dialect speaker

FS2
FS2-AP
FS2-AP-Scratch
FS2-AP-W/O
FS2-REF
FS2-REF-F0
REF

CD-TTS: Synthesis of Osaka-dialect speech by Tokyo-dialect speaker

FS2
FS2-AP
FS2-AP-Scratch
FS2-AP-W/O
FS2-REF
FS2-REF-F0
REF

2. Samples of LLM-based dialect translation

We present text samples translated into Osaka-dialect using Swallow 13B. Original sentences written in standard Japanese are sampled from ReazonSpeech corpus.

Original sentence written in standard Japanese	Sentence translated into Osaka-dialect
本当だって思えばいいって事ですよね?	ほんまやと思えばええんちゃうか?
北日本では大雪となるおそれもありますので、今のうちに備えをしておいてください。	北日本では大雪になるかもしれへんから、今のうちに備えといた方がええで。
人間にしかできないことはなんなのか考えてみるといいね。	人間にしかできひんことはなんなのか考えてみたらええね。
それでもうひたすらそれを着々とやってるっていう、そういう前提で考えるべきだと思います。	それでもうひたすらそれを着々とやってるっていう、そういう前提で考えるべきやと思う。
多くの相撲ファンは同じ気持ちかもしれないですね。	多くの相撲ファンは同じ気持ちかもしれへんね。
出火当時女性は家にいたとみられますが現在連絡がとれていないということです。	出火当時女性は家におったみたいやけど、現在連絡がとれへんみたいやで。
今日は風船で遊びます！	今日は風船で遊ぶで！
あの時僕はもうお金もないですし、一張羅の時計ですよ。	あの時僕はもうお金もないし、一張羅の時計やねん。
とことん遊ぶって大事ですね。	とことん遊ぶって大事やな。
あっそう、便秘の女性にもすごくいいと思います。	あっそうや、便秘の女性にもめっちゃええと思うで。

References

Y. Ren, C. Hu, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T.-Y. Liu, "FastSpeech 2: Fast and high-quality end-to-end text to speech," in Proc. ICLR, 2021.
Y. A. Li, C. Han, X. Jiang, and N. Mesgarani, "Phoneme-level BERT for enhanced prosody of text-to-speech with grapheme predictions," in Proc. ICASSP, 2023, pp. 1-5.
S. Takamichi and H. Saruwatari, "CPJD corpus: Crowdsourced parallel speech corpus of japanese dialects," in Proc. LREC, 2018, pp. 434-437.
S. Takamichi and H. Saruwatari, "JMD: Japanese multi-dialect corpus," (2021). [Online]. Available: https://sites.google.com/site/shinnosuketakamichi/research-topics/jmd_corpus?authuser=0
S. Takamichi, R. Sonobe, K. Mitsui, Y. Saito, T. Koriyama, N. Tanji, and H. Saruwatari, "JSUT and JVS: Free Japanese voice corpora for accelerating speech synthesis research," Acoustical Science and Technology, vol. 41, no. 5, pp. 761-768, Sep. 2020.