Availability and prerequisitesedit

This connector is available as a native connector in Elastic versions 8.5.0 and later.

To use this connector natively in Elastic Cloud, satisfy all native connector requirements.

Compatibilityedit

This connector is compatible with MongoDB Atlas and MongoDB 3.6 and later.

The data source and your Elastic deployment must be able to communicate with each other over a network.

Configurationedit

Each time you create an index to be managed by this connector, you will create a new connector configuration. You will need some or all of the following information about the data source.

Create a MongoDB connectoredit

Use the UIedit

To create a new MongoDB connector:

For additional operations, see Using connectors.

Use the APIedit

You can use the Elasticsearch Create connector API to create a new native MongoDB connector.

For example:

PUT _connector/<my-mongo-connector>
{
  "index_name": "<my-elasticsearch-index>",
  "name": "Content synced from MongoDB",
  "service_type": "mongo",
  "is_native": "true"
}

Refer to the Elasticsearch API documentation for details of all available Connector APIs.

Usageedit

To use this connector as a native connector, use the Connector workflow. See Native connectors (managed service).

For additional operations, see Using connectors.

Exampleedit

An example is available for this connector. See Native connector tutorial (MongoDB).

Known issuesedit

SSL must be enabled for MongoDB Atlasedit

Expressions and variables in aggregation pipelinesedit

It’s not possible to use expressions like new Date() inside an aggregation pipeline. These expressions won’t be evaluated by the underlying MongoDB client, but will be passed as a string to the MongoDB instance. A possible workaround is to use aggregation variables.

Incorrect (new Date() will be interpreted as string):

{
    "aggregate": {
        "pipeline": [
            {
                "$match": {
                  "expiresAt": {
                    "$gte": "new Date()"
                  }
                }
            }
        ]
    }
}

Correct (usage of $$NOW):

{
  "aggregate": {
    "pipeline": [
      {
        "$addFields": {
          "current_date": {
            "$toDate": "$$NOW"
          }
        }
      },
      {
        "$match": {
          "$expr": {
            "$gte": [
              "$expiresAt",
              "$current_date"
            ]
          }
        }
      }
    ]
  }
}

Connecting with self-signed or custom CA TLS Certedit

Currently, the MongoDB connector does not support working with self-signed or custom CA certs when connecting to your self-managed MongoDB host.

This can be worked around in development environments, by appending certain query parameters to the configured host.

For example, if your host is mongodb+srv://my.mongo.host.com, appending ?tls=true&tlsAllowInvalidCertificates=true will allow disabling TLS certificate verification.

The full host in this example will look like this:

mongodb+srv://my.mongo.host.com/?tls=true&tlsAllowInvalidCertificates=true

See Known issues for any issues affecting all connectors.

Troubleshootingedit

See Troubleshooting.

Securityedit

See Security.

Documents and syncsedit

The following describes the default syncing behavior for this connector. Use sync rules and ingest pipelines to customize syncing for specific indices.

All documents in the configured MongoDB database and collection are extracted and transformed into documents in your Elasticsearch index.

This results in Elasticsearch documents that closely match the original MongoDB documents.

The Elasticsearch mapping is created when the first document is created.

Each sync is a "full" sync. For each MongoDB document discovered:

This is recursive, because embedded documents can themselves contain embedded documents.

Sync rulesedit

The following sections describe Sync rules for this connector.

Basic sync rules are identical for all connectors and are available by default.

Advanced rules for MongoDB can be used to express either find queries or aggregation pipelines. They can also be used to tune options available when issuing these queries/pipelines.

find queriesedit

For find queries, the structure of this JSON DSL should look like:

{
	"find":{
		"filter": {
			// find query goes here
		},
		"options":{
			// query options go here
		}
	}
}

For example:

{
	"find": {
		"filter": {
			"$text": {
				"$search": "garden",
				"$caseSensitive": false
			}
		},
		"skip": 10,
		"limit": 1000
	}
}

find queries also support additional options, for example the projection object:

{
  "find": {
    "filter": {
      "languages": [
        "English"
      ],
      "runtime": {
        "$gt":90
      }
    },
    "projection":{
      "tomatoes": 1
    }
  }
}

Where the available options are:

Aggregation pipelinesedit

Similarly, for aggregation pipelines, the structure of the JSON DSL should look like:

{
	"aggregate":{
		"pipeline": [
			// pipeline elements go here
		],
		"options": {
            // pipeline options go here
		}
    }
}

Where the available options are:

Content extractionedit

See Content extraction.

Migrating from the Ruby connector frameworkedit

As part of the 8.8.0 release the MongoDB connector was moved from the Ruby connectors framework to the Elastic connector framework.

This change introduces minor formatting modifications to data ingested from MongoDB:

If your MongoDB connector stopped working after migrating from 8.7.x to 8.8.x, read the workaround outlined in Known issues. If that does not work, we recommend deleting the search index attached to this connector and re-creating a MongoDB connector from scratch.

Availability and prerequisitesedit

This connector is also available as a connector client from the Elastic connector framework. To use this connector as a connector client, satisfy all connector client requirements.

Compatibilityedit

This connector is compatible with MongoDB Atlas and MongoDB 3.6 and later.

The data source and your Elastic deployment must be able to communicate with each other over a network.

Configurationedit

The following configuration fields are required to set up the connector:

Create a MongoDB connectoredit

Use the UIedit

To create a new MongoDB connector:

For additional operations, see Using connectors.

Use the APIedit

You can use the Elasticsearch Create connector API to create a new self-managed MongoDB connector client.

For example:

PUT _connector/my-mongo-connector
{
  "index_name": "my-elasticsearch-index",
  "name": "Content synced from MongoDB",
  "service_type": "mongo"
}

Refer to the Elasticsearch API documentation for details of all available Connector APIs.

Usageedit

To use this connector as a connector client, see Connector clients (self-managed) For additional usage operations, see Using connectors.

Exampleedit

An example is available for this connector. See Native connector tutorial (MongoDB).

Known issuesedit

SSL must be enabled for MongoDB Atlasedit

Expressions and variables in aggregation pipelinesedit

It’s not possible to use expressions like new Date() inside an aggregation pipeline. These expressions won’t be evaluated by the underlying MongoDB client, but will be passed as a string to the MongoDB instance. A possible workaround is to use aggregation variables.

Incorrect (new Date() will be interpreted as string):

{
    "aggregate": {
        "pipeline": [
            {
                "$match": {
                  "expiresAt": {
                    "$gte": "new Date()"
                  }
                }
            }
        ]
    }
}

Correct (usage of $$NOW):

{
  "aggregate": {
    "pipeline": [
      {
        "$addFields": {
          "current_date": {
            "$toDate": "$$NOW"
          }
        }
      },
      {
        "$match": {
          "$expr": {
            "$gte": [
              "$expiresAt",
              "$current_date"
            ]
          }
        }
      }
    ]
  }
}

Connecting with self-signed or custom CA TLS Certedit

Currently, the MongoDB connector does not support working with self-signed or custom CA certs when connecting to your self-managed MongoDB host.

This can be worked around in development environments, by appending certain query parameters to the configured host.

For example, if your host is mongodb+srv://my.mongo.host.com, appending ?tls=true&tlsAllowInvalidCertificates=true will allow disabling TLS certificate verification.

The full host in this example will look like this:

mongodb+srv://my.mongo.host.com/?tls=true&tlsAllowInvalidCertificates=true

Docker image errors out for versions 8.12.0 and 8.12.1edit

A bug introduced in 8.12.0 causes the Connectors docker image to error out if run using MongoDB as its source. The command line will output the error cannot import name 'coroutine' from 'asyncio'. This issue is fixed in versions 8.12.2 and 8.13.0. This bug does not affect native connectors.

See Known issues for any issues affecting all connectors.

Troubleshootingedit

See Troubleshooting.

Securityedit

See Security.

Deployment using Dockeredit

You can deploy the MongoDB connector as a self-managed connector client using Docker. Follow these instructions.

curl https://raw.githubusercontent.com/elastic/connectors/main/config.yml.example --output ~/connectors-config/config.yml

# When connecting to your cloud deployment you should edit the host value
elasticsearch.host: http://host.docker.internal:9200
elasticsearch.api_key: <ELASTICSEARCH_API_KEY>

connectors:
  -
    connector_id: <CONNECTOR_ID_FROM_KIBANA>
    service_type: mongo
    api_key: <CONNECTOR_API_KEY_FROM_KIBANA> # Optional. If not provided, the connector will use the elasticsearch.api_key instead

docker run \
-v ~/connectors-config:/config \
--network "elastic" \
--tty \
--rm \
docker.elastic.co/enterprise-search/elastic-connectors:8.13.2.0 \
/app/bin/elastic-ingest \
-c /config/config.yml

Refer to DOCKER.md in the elastic/connectors repo for more details.

Find all available Docker images in the official registry.

Documents and syncsedit

The following describes the default syncing behavior for this connector. Use sync rules and ingest pipelines to customize syncing for specific indices.

All documents in the configured MongoDB database and collection are extracted and transformed into documents in your Elasticsearch index.

This results in Elasticsearch documents that closely match the original MongoDB documents.

The Elasticsearch mapping is created when the first document is created.

Each sync is a "full" sync. For each MongoDB document discovered:

This is recursive, because embedded documents can themselves contain embedded documents.

Sync rulesedit

The following sections describe Sync rules for this connector.

Basic sync rules are identical for all connectors and are available by default.

Advanced rules for MongoDB can be used to express either find queries or aggregation pipelines. They can also be used to tune options available when issuing these queries/pipelines.

find queriesedit

For find queries, the structure of this JSON DSL should look like:

{
	"find":{
		"filter": {
			// find query goes here
		},
		"options":{
			// query options go here
		}
	}
}

For example:

{
	"find": {
		"filter": {
			"$text": {
				"$search": "garden",
				"$caseSensitive": false
			}
		},
		"skip": 10,
		"limit": 1000
	}
}

find queries also support additional options, for example the projection object:

{
  "find": {
    "filter": {
      "languages": [
        "English"
      ],
      "runtime": {
        "$gt":90
      }
    },
    "projection":{
      "tomatoes": 1
    }
  }
}

Where the available options are:

Aggregation pipelinesedit

Similarly, for aggregation pipelines, the structure of the JSON DSL should look like:

{
	"aggregate":{
		"pipeline": [
			// pipeline elements go here
		],
		"options": {
            // pipeline options go here
		}
    }
}

Where the available options are:

Content extractionedit

See Content extraction.

Migrating from the Ruby connector frameworkedit

As part of the 8.8.0 release the MongoDB connector was moved from the Ruby connectors framework to the Elastic connector framework.

This change introduces minor formatting modifications to data ingested from MongoDB:

If your MongoDB connector stopped working after migrating from 8.7.x to 8.8.x, read the workaround outlined in Known issues. If that does not work, we recommend deleting the search index attached to this connector and re-creating a MongoDB connector from scratch.