> For the complete documentation index, see [llms.txt](https://dzhelezov.gitbook.io/hydra/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://dzhelezov.gitbook.io/hydra/hydra-docs/hydra-indexer.md).

# Hydra Indexer

Hydra Indexer is a daemon that ingests raw substrate data such as events and extrinsics from a substrate chain and saves it into a Postgres database. It is typically used in conjunction with [Hydra Indexer Gateway](/hydra/hydra-docs/hydra-indexer-gateway.md) providing a GraphQL API interface.

## Installation

The indexer exits if an unrecoverable error occurs. It is therefore advisable to run the indexer with a process manager, such as [PM2](https://pm2.keymetrics.io/) or Docker.

### Prerequisites

* Node 12.x
* Postgres database
* Redis instances
* (Optional) Docker

### Environment variables

The indexer is set up using the following environment variables

| Variable                    | Default | Required |                                                                             Description |
| --------------------------- | :-----: | -------: | --------------------------------------------------------------------------------------: |
| WS\_PROVIDER\_ENDPOINT\_URI |    -    |  **Yes** |                                                       Substrate API endpoint to connect |
| REDIS\_URI                  |    -    |  **Yes** |                                                                      Redis instance URI |
| DB\_NAME                    |    -    |  **Yes** |                                                                           Database name |
| DB\_PORT                    |    -    |  **Yes** |                                                                           Database port |
| DB\_HOST                    |    -    |  **Yes** |                                                                           Database host |
| DB\_USER                    |    -    |  **Yes** |                                                                           Database user |
| DB\_PASS                    |    -    |  **Yes** |                                                                       Database password |
| TYPES\_JSON                 |    -    |       No |                              Path to a JSON type definition with custom Substrate types |
| BLOCK\_HEIGHT               |    0    |       No | Block height to start indexing. Ignored if the database already contains indexed blocks |

### Manual setup

Run

```bash
yarn && yarn build
```

For setting up the database and running the migrations, run `yarn db:bootstrap`. Make sure the environment variables `DB_*` are set.

For starting the indexer, run `yarn start:prod`.

### Docker image

There are pre-built runnable docker images in `joystream/hydra-indexer` docker repo. The default command is `yarn start:prod`.

First, bootstrap the database:

```bash
docker run -e DB_HOST=... -e DB_PORT=... -e DB_NAME=... -e DB_PASS=... -e DB_USER=... joystream/hydra-indexer sh -c 'yarn db:bootstrap'
```

Then run the indexer (make sure that all the required environment variables are set)

```bash
docker run -e ... joystream/hydra-indexer
```

## Advanced environment variables

Some optional environment variables are available for fine-tuning.

| Variable                        |      Default     |                                                                                            Description |
| ------------------------------- | :--------------: | -----------------------------------------------------------------------------------------------------: |
| BLOCK\_CACHE\_TTL\_SEC          |      `60*60`     |                                                            TTL for processed blocks in the Redis cache |
| INDEXER\_HEAD\_TTL\_SEC         |      `60*15`     |                                                                   TTL for the indexer head block entry |
| WORKERS\_NUMBER                 |         5        |                                                       Number of concurrent workers fetching the blocks |
| BLOCK\_PRODUCER\_FETCH\_RETRIES |         3        | Number of attempts fetching each a block before throwing an error. Set to `-1` for indefinite attempts |
| SUBSTRATE\_API\_TIMEOUT         |  `1000 * 60 * 5` |                                                                Timeout in (milliseconds) for API calls |
| NEW\_BLOCK\_TIMEOUT\_MS         | `60 * 10 * 1000` |                                      Panic if no blockchain blocks have been received within this time |