Document Management

This section explains the feature for registering reference documents. By setting the content of the registered documents during chat creation, you can have the AI generate answers that take the document content into account.

To reference documents from chat, you need to register the files and create a retriever. The retriever stores fragments of registered files and data for searching.

By setting the retriever during chat creation, you can generate AI answers that take the content of the registered documents into account.

Available File Formats

The following file formats can be used:

• Text documents (.txt)
• PDF files (.pdf)
• Microsoft Word documents (.docx)
• Microsoft PowerPoint presentations (.pptx)
• HTML (.html)
• Markdown files (.md)
• CSV files (.csv)
• Microsoft Excel documents (.xlsx)

Warning

CSV files and Excel documents are limited to specific formats.

Specifically, the file should have column titles in the first row and data in the following rows. This functionality is designed for manually splitting files during retriever registration and registering them in the retriever. It does not guarantee successful use for other purposes.

Please note that registering data as a general table may not work well, as it is primarily intended for retriever registration.

File Registration

Click on 'Data Processing' in the bottom left corner of the sidebar to navigate to the document management page.

The document management page will be displayed.

Click 'Add Documents' to display the upload window.

Use the file uploader to select the documents you want to register.

The selected documents will be displayed.

You can optionally change the document name (if not set, the original file name will be used). You can also register multiple files at once. Please select multiple files in the uploader.

Click the "Upload" button to register the files.

By checking the "Create Retriever" option, you can simultaneously create a retriever for the registered files. For more details on this operation, please refer to the next section, "Retriever Registration".

Retriever Registration

To register files to a retriever, first check the checkboxes of the files you want to register to the retriever. Next, click the "Register to Retriever" button to display the retriever registration modal. In the displayed modal, select the retriever and click "Submit" to register the files to the retriever.

Additionally, you can create a new retriever by entering a new retriever name and registering it simultaneously.

When creating a retriever, you can also utilize the keyword mask feature. For more details on the keyword masking feature, please refer to the keyword mask page.

Once the registration is complete, the registered files will be displayed in the "File List" tab, and the retriever will be displayed in the "Retriever List" tab.

If you see an Internal Error

The registration of retrievers is done using the Azure OpenAI embedding API. If you register a large number of files at once or large size file, you may encounter errors which says "Internal Server Error". There may be cases where the rate limit on the Azure OpenAI is the cause.

If the registration of the retriever fails, please try again after some time or consider splitting large files and adding them to an existing retriever.

Creating a Document from a URL

You can create a document from a specified URL.

Click the 'Add Url' button.

Specify the URL. Optionally, give the document a name (if not specified, the URL will be used as the name).

Click the 'Create' button to register the document.

The process for registering a retriever is the same as registering a file.

Note

Only static websites can be registered. Please note that registering websites that dynamically generate pages using JavaScript or similar technologies may not work properly.

Document List and Deletion

You can check the registered files and retrievers in the "File List" and "Retriever List" tabs, respectively.

To delete a registered file or retriever, click on the trash can icon.

Even if you delete a file, the content of that file will not be lost from the retriever that includes it.

Deleting a retriever will prevent any chat that references it from continuing the conversation. However, you can still access the conversation history.