# Data Ingestion Process

After completing the installation, follow these steps to index all contents related to a specific use case:

### Release 3.0.0

1. Install Python on the machine where the files need to be ingested.
2. Clone Git Repo from [https://github.com/Sunbird-AIAssistant/sakhi-api-service](https://www.google.com/url?q=https://github.com/Sunbird-AIAssistant/sakhi-api-service\&sa=D\&source=editors\&ust=1712142234872409\&usg=AOvVaw23d0wT4lcVdCV9qqQUVOq8).
3. Go to the root directory and update the `.env` file with the necessary [vector store configuration](https://ai-assistant.sunbird.org/components/sakhi-api-service/environment-variables) values. 
4. Run the following: 

{% code overflow="wrap" %}

```python
Step 1: pip install -r requirements-dev.txt 
Step 2: python3 index_documents.py --folder_path=<PATH_TO_INPUT_FILE_DIRECTORY> --fresh_index --chunk_size=1024 --chunk_overlap=100

# --fresh_index: Create a new index from scratch.
# --chunk_size: Divide the documents into chunks of 1024 characters. Default: 1024
# --chunk_overlap: Overlap each chunk by 100 characters for context. Default: 100
```

{% endcode %}

### Before Release 3.0.0

1. Install Python on the machine where the files need to be ingested.
2. Place the files to be indexed in a folder on the machine.
3. Download index\_documents.py and requirements-dev.txt file from [https://github.com/Sunbird-AIAssistant/sakhi-api-service](https://www.google.com/url?q=https://github.com/Sunbird-AIAssistant/sakhi-api-service\&sa=D\&source=editors\&ust=1712142234872409\&usg=AOvVaw23d0wT4lcVdCV9qqQUVOq8)
4. Run the following:&#x20;

<pre class="language-python" data-overflow="wrap"><code class="lang-python"><strong>Step 1: pip install -r requirements-dev.txt 
</strong><strong>Step 2: python3 index_documents.py --marqo_url=&#x3C;MARQO_URL> --index_name=&#x3C;MARQO_INDEX_NAME> --folder_path=&#x3C;PATH_TO_INPUT_FILE_DIRECTORY> --fresh_index
</strong></code></pre>

**Notes**:

1. Please run the commands via screen background, as it will take a couple of hours to run
2. “--fresh\_index” is to be used when you run the indexing for the first time or delete the existing index and freshly index it. If you want to append new files to the existing index, run it without --fresh\_index
3. For running without --fresh\_index, ensure your new files are kept in a new folder and the --folder\_path is pointed to only the new files.