～ Chunkとおさらば！（とはいかない・・・）～

QEU:FOUNDER ： “今回も前回に引き続き、Cohere ならではのRAG(Retrieval Argument Genera-tion)の話をします。ネタは前回のUR紛争のネタからとってきましょう。”

QEU:FOUNDER ： “以下の文章からRAGを取る場合にはどうしますか？”

FACT 1: Historical Ties According to Wikipedia, "The Russian Orthodox Church has its roots in the Kievan Rus', a medieval state that existed from the 9th to the 13th centuries. The Church was founded in 988 by Prince Vla-dimir the Great, who converted to Christianity." (Source: Wikipedia, "Russian Orthodox Church")

FACT 2: Ukrainian Orthodox Church's Autonomy In an article titled "Ukraine's Orthodox Church gains independence from Russia" published in The Guardian, it is stated that "The Ukrainian Orthodox church has been granted independence from Russia, a move that could strengthen Ukrainian identity and weaken Russian influence." (Source: The Guardian, "Ukraine's Orthodox Church gains independence from Russia")

FACT 3: Russian Orthodox Church's Influence on Russian Politics In the book "Russian Politics and Society" by Richard Sakwa, it is written that "The Russian Ortho-dox Church has played a significant role in shaping Russian identity and has been closely tied to the Russian state throughout its history." (Source: Richard Sakwa, "Russian Politics and Society", Routledge, 2017)

FACT 4: Ukrainian Orthodox Church's Distinct Identity In an interview with Metropolitan Epiphaniy, the head of the Orthodox Church of Ukraine, he stated that "The Ukrainian Orthodox Church has its own distinct identity and traditions, which are sepa-rate from the Russian Orthodox Church." (Source: Interview with Metropolitan Epiphaniy, "Ukrainian Orthodox Church gains independence from Russia", The Guardian)

FACT 5: Historical Differences in Orthodox Christianity A proverb from Eastern Orthodox Christianity states, "Orthodoxy is not a religion, it's a way of life." This highlights the deep cultural and historical roots of Orthodox Christianity in both Russia and Ukraine, but also underscores the distinct ways in which it has shaped the social and political landscape of each country.

D先生： “上の文章を「Chunk」という文章単位に分割します。普通は256wordsから512wordsの文字列長さあたりですかね・・・。”

QEU:FOUNDER ： “もともとFACT_NOで分割されているので、Chunkを使わなくてもいいんじゃないですか？プロンプトエンジニアリングを使えば・・・。”

Read the following text (TEXT), split it into FACTs, enclose the strings in brackets ('[]'), concatenate them, and convert them into a list format of '["[FACT 1]", "[FACT 2]", ..., "[FACT 5]"]'
TEXT```
FACT 1: Historical Ties
According to Wikipedia, "The Russian Orthodox Church has its roots in the Kievan Rus', a medieval state that existed from the 9th to the 13th centuries. The Church was founded in 988 by Prince Vla-dimir the Great, who converted to Christianity." (Source: Wikipedia, "Russian Orthodox Church")

FACT 2: Ukrainian Orthodox Church's Autonomy
In an article titled "Ukraine's Orthodox Church gains independence from Russia" published in The Guardian, it is stated that "The Ukrainian Orthodox church has been granted independence from Russia, a move that could strengthen Ukrainian identity and weaken Russian influence." (Source: The Guardian, "Ukraine's Orthodox Church gains independence from Russia")

FACT 3: Russian Orthodox Church's Influence on Russian Politics
In the book "Russian Politics and Society" by Richard Sakwa, it is written that "The Russian Ortho-dox Church has played a significant role in shaping Russian identity and has been closely tied to the Russian state throughout its history." (Source: Richard Sakwa, "Russian Politics and Society", Routledge, 2017)

FACT 4: Ukrainian Orthodox Church's Distinct Identity
In an interview with Metropolitan Epiphaniy, the head of the Orthodox Church of Ukraine, he stated that "The Ukrainian Orthodox Church has its own distinct identity and traditions, which are sepa-rate from the Russian Orthodox Church." (Source: Interview with Metropolitan Epiphaniy, "Ukrainian Orthodox Church gains independence from Russia", The Guardian)

FACT 5: Historical Differences in Orthodox Christianity
A proverb from Eastern Orthodox Christianity states, "Orthodoxy is not a religion, it's a way of life." This highlights the deep cultural and historical roots of Orthodox Christianity in both Russia and Ukraine, but also underscores the distinct ways in which it has shaped the social and political landscape of each country.
```

D先生： “このような手も考えられますよね。でも、Chunkを使わずにRAGは使えるんですか？”

QEU:FOUNDER ： “ちょっと実験をしてみましょう。以下のような、EXCELのフォーマットファイルを作成して、読み込みます。”

QEU:FOUNDER ： “それではプログラムをドン。”

# -----
# 必要なライブラリのインポート
# -----
import os, re, time
import pandas as pd
import json

# Excelファイルからデータフレームを作成
df = pd.read_excel('document_format.xlsx')
df

# ---
for i in range(len(df)):

    str_fact = arr_facts[i]
    str_fact = str_fact.replace("[","")
    str_fact = str_fact.replace("]","")
    df.loc[i,"snippet"] = str_fact
    print(str_fact)


# データフレームをJSONに変換
json_result = df.to_json(orient='records')

# JSON文字列を整形して表示
parsed = json.loads(json_result)

QEU:FOUNDER ： “変数「parsed」を見てみましょう。”

D先生： “さっきの文章を５つに分割し、JSON型式にしているわけですね。”

QEU:FOUNDER ： “この情報をCohere特製のRAGに入れましょう。”

# -----
# 必要なライブラリのインポート
# -----
import os, re, time
import pandas as pd
import json

# Excelファイルからデータフレームを作成
df = pd.read_excel('document_format.xlsx')
df

# ---
for i in range(len(df)):

    str_fact = arr_facts[i]
    str_fact = str_fact.replace("[","")
    str_fact = str_fact.replace("]","")
    df.loc[i,"snippet"] = str_fact
    print(str_fact)

# データフレームをJSONに変換
json_result = df.to_json(orient='records')

# JSON文字列を整形して表示
parsed = json.loads(json_result)
#parsed_json = json.dumps(parsed, indent=2)
#print(parsed_json)

# -----
# EXAMPLE
import cohere
#co = cohere.Client(api_key="<YOUR API KEY>")
co = cohere.Client()

response = co.chat(
  #model="command-r-plus",
  model="command-r",
  message="In what ways did the influence of Orthodox Christianity and the Russian Orthodox Church shape the social and political landscape of Russia, and how did this differ from its impact on Ukraine?",
  documents=parsed)
print(response)

QEU:FOUNDER ： “responseの中には、多量のデータが詰まっています。”

D先生： “回答そのものを見せてもらえますか？”

print(response.text)
The Russian Orthodox Church has profoundly influenced Russia's social and political landscape throughout its history. It has played a pivotal role in shaping the Russian identity, often operating in close conjunction with the state. This symbiotic relationship between church and state has helped de-fine Russia's cultural and spiritual values, constituting a central pillar of Russian life.

The situation in Ukraine differs significantly. While Ukraine also has a long-standing Orthodox Christian tradition, the Ukrainian Orthodox Church maintains its own distinct identity and practices, separate from the Russian Orthodox Church. In a move to reinforce Ukrainian independence and national identity, the Ukrainian Orthodox Church has been granted independence from Russia. This separation may help to diminish Russian influence within Ukraine, contributing to a distinct social and political dynamic compared to Russia.

Both Russia and Ukraine share a rich Orthodox Christian heritage, but the influence of the Russian Orthodox Church and its interaction with the state has been a key differentiator between the two na-tions.
ロシア正教会は、ロシアの歴史を通じて、その社会的、政治的状況に多大な影響を与えてきました。ロシア正教会は、しばしば国家と密接に連携しながら、ロシアのアイデンティティを形成する上で極めて重要な役割を果たしてきました。教会と国家のこの共生関係は、ロシアの文化的、精神的価値を定義するのに役立ち、ロシア生活の中心的な柱となっています。

ウクライナの状況は大きく異なります。ウクライナにも長年にわたる正教会の伝統がありますが、ウクライナ正教会はロシア正教会とは別に、独自のアイデンティティと慣習を維持しています。ウクライナの独立と国家的アイデンティティを強化する動きの中で、ウクライナ正教会はロシアからの独立を認められました。この分離は、ウクライナ内でのロシアの影響力を弱めるのに役立ち、ロシアとは異なる社会的、政治的ダイナミクスに貢献する可能性があります。

ロシアとウクライナはどちらも豊かな正教会の伝統を共有していますが、ロシア正教会の影響と国家との関わりが、両国間の重要な差別化要因となっています。

QEU:FOUNDER ： “まあ、普通の出力結果ですね。Command-rほどの規模のLLMであれば、楽勝でしょう。”

D先生： “前回もいいましたが、FACTはLLMから抽出してきたものですから、このRAGそのものが意味があるのかなあ？”

QEU:FOUNDER ： “このFACT群がWeb-searchから得られているのモノであれば、大いにいみがあるでしょう？ついでに、我々が付与した文書群の引用状況をチェックしてみましょう。”

print(response.citations) [ChatCitation(start=32, end=93, text="profoundly influenced Russia's social and political landscape", document_ids=['doc_2']), ChatCitation(start=109, end=117, text='history.', document_ids=['doc_2']), ChatCitation(start=134, end=178, text='pivotal role in shaping the Russian identity', docu-ment_ids=['doc_2']), ChatCitation(start=199, end=232, text='close conjunction with the state.', doc-ument_ids=['doc_2']), ChatCitation(start=238, end=260, text='symbiotic relationship', docu-ment_ids=['doc_2']), ChatCitation(start=297, end=342, text="define Russia's cultural and spiritual values", document_ids=['doc_2']), ChatCitation(start=359, end=390, text='central pillar of Russian life.', document_ids=['doc_2']), ChatCitation(start=466, end=508, text='long-standing Orthodox Christian tradition', document_ids=['doc_0', 'doc_3']), ChatCitation(start=514, end=539, text='Ukrainian Orthodox Church', document_ids=['doc_3']), ChatCitation(start=554, end=589, text='own distinct identity and practices', document_ids=['doc_3']), ChatCitation(start=591, end=633, text='separate from the Russian Orthodox Church.', document_ids=['doc_3']), ChatCita-tion(start=647, end=679, text='reinforce Ukrainian independence', document_ids=['doc_1']), ChatCitation(start=684, end=701, text='national identity', document_ids=['doc_1']), ChatCita-tion(start=742, end=775, text='granted independence from Russia.', document_ids=['doc_1']), ChatCitation(start=781, end=791, text='separation', document_ids=['doc_1']), ChatCita-tion(start=804, end=830, text='diminish Russian influence', document_ids=['doc_1']), ChatCita-tion(start=962, end=989, text='Orthodox Christian heritage', document_ids=['doc_0', 'doc_3']), ChatCitation(start=999, end=1039, text='influence of the Russian Orthodox Church', docu-ment_ids=['doc_2', 'doc_3']), ChatCitation(start=1048, end=1074, text='interaction with the state', document_ids=['doc_2']), ChatCitation(start=1086, end=1104, text='key differentiator', docu-ment_ids=['doc_1', 'doc_2', 'doc_3'])]

QEU:FOUNDER ： “この情報に、引用がどのように行われたのかが記述されています。”