Data Ingestion Process

After completing the installation, follow these steps to index all contents related to a specific use case:

Release 3.0.0

1. Install Python on the machine where the files need to be ingested.

2. Clone Git Repo from https://github.com/Sunbird-AIAssistant/sakhi-api-service.

3. Go to the root directory and update the .env file with the necessary vector store configuration values.

4. Run the following:

Step 1: pip install -r requirements-dev.txt 
Step 2: python3 index_documents.py --folder_path=<PATH_TO_INPUT_FILE_DIRECTORY> --fresh_index --chunk_size=1024 --chunk_overlap=100

# --fresh_index: Create a new index from scratch.
# --chunk_size: Divide the documents into chunks of 1024 characters. Default: 1024
# --chunk_overlap: Overlap each chunk by 100 characters for context. Default: 100

Before Release 3.0.0

1. Install Python on the machine where the files need to be ingested.

2. Place the files to be indexed in a folder on the machine.

3. Download index_documents.py and requirements-dev.txt file from https://github.com/Sunbird-AIAssistant/sakhi-api-service

4. Run the following:

Step 1: pip install -r requirements-dev.txt 
Step 2: python3 index_documents.py --marqo_url=<MARQO_URL> --index_name=<MARQO_INDEX_NAME> --folder_path=<PATH_TO_INPUT_FILE_DIRECTORY> --fresh_index

Notes:

1. Please run the commands via screen background, as it will take a couple of hours to run

2. “--fresh_index” is to be used when you run the indexing for the first time or delete the existing index and freshly index it. If you want to append new files to the existing index, run it without --fresh_index

3. For running without --fresh_index, ensure your new files are kept in a new folder and the --folder_path is pointed to only the new files.