A powerful, easily deployable network traffic analysis tool suite for network security monitoring
OpenSearch and Elasticsearch instances
Malcolm’s default standalone configuration is to use a local OpenSearch instance in a container to index and search network traffic metadata. OpenSearch can also run as a cluster with instances distributed across multiple nodes with dedicated roles such as cluster manager, data node, ingest node, etc.
As the permutations of OpenSearch cluster configurations are numerous, it is beyond Malcolm’s scope to set up multi-node clusters. However, Malcolm can be configured to use a remote OpenSearch cluster rather than its own internal instance.
As an alternative to OpenSearch, Malcolm may now be configured to use a remote Elasticsearch cluster with its own instance of Kibana. This configuration is intended for users that already have the Elastic stack deployed in their environments; OpenSearch is recommended for greenfield deployments.
The OPENSEARCH_… environment variables in opensearch.env control whether Malcolm uses its own local OpenSearch instance (opensearch-local), a remote OpenSearch instance (opensearch-remote) or a remote Elasticsearch instance (elasticsearch-remote) as its primary data store. The configuration portion of Malcolm install script (./scripts/configure) can help users configure these options.
For example, to use the default standalone configuration, answer Y when prompted Should Malcolm use and maintain its own OpenSearch instance?
To use a remote OpenSearch cluster:
…
Should Malcolm use and maintain its own OpenSearch instance? (Y / n): n
1: opensearch-local - local OpenSearch
2: opensearch-remote - remote OpenSearch
3: elasticsearch-remote - remote Elasticsearch
Select primary Malcolm document store (opensearch-local): 2
Enter primary remote OpenSearch connection URL (e.g., https://192.168.1.123:9200): https://10.9.0.215:9200
Require SSL certificate validation for communication with remote OpenSearch instance? (y / N): n
You must run auth_setup after configure to store data store connection credentials.
…
To use a remote Elasticsearch cluster and Kibana:
…
Should Malcolm use and maintain its own OpenSearch instance? (Y / n): n
1: opensearch-local - local OpenSearch
2: opensearch-remote - remote OpenSearch
3: elasticsearch-remote - remote Elasticsearch
Select primary Malcolm document store (opensearch-local): 3
Enter primary remote Elasticsearch connection URL (e.g., https://192.168.1.123:9200) (): https://10.9.0.215:9200
Require SSL certificate validation for communication with remote Elasticsearch instance? (y / N): n
Enter Kibana connection URL (e.g., https://192.168.1.123:5601) (): https://10.9.0.215:5601
You must run auth_setup after configure to store data store connection credentials.
…
Whether the primary data store is a locally maintained single-node instance or is a remote cluster, Malcolm can additionally be configured to forward logs to a secondary remote OpenSearch or Elasticsearch instance. The OPENSEARCH_SECONDARY_… environment variables in opensearch.env control this behavior. Configuration of a remote secondary data store is similar to that of a remote primary data store:
…
Forward Logstash logs to a secondary remote document store? (y / N): y
1: opensearch-remote - remote OpenSearch
2: elasticsearch-remote - remote Elasticsearch
Select secondary Malcolm document store: 1
Enter secondary remote OpenSearch connection URL (e.g., https://192.168.1.123:9200) (): https://10.9.0.216:9200
Require SSL certificate validation for communication with secondary remote OpenSearch instance? (y / N): n
You must run auth_setup after configure to store data store connection credentials.
…
Authentication and authorization for remote data store clusters
In addition to setting the environment variables in opensearch.env as described above, users must provide Malcolm with credentials for it to communicate with remote OpenSearch and Elasticsearch instances. These credentials are stored in the Malcolm installation directory as .opensearch.primary.curlrc and .opensearch.secondary.curlrc for the primary and secondary data store connections, respectively, and are bind-mounted into the containers that need to communicate with OpenSearch/Elasticsearch. These cURL-formatted config files can be generated for you by the auth_setup script as illustrated:
$ ./scripts/auth_setup
…
Store username/password for primary remote OpenSearch/Elasticsearch instance? (y / N): y
OpenSearch/Elasticsearch username: servicedb
servicedb password:
servicedb password (again):
Additional local accounts can be created at https://localhost/auth/ when Malcolm is running
Require SSL certificate validation for OpenSearch/Elasticsearch communication? (Y / n): n
Will Malcolm be using an existing remote primary or secondary OpenSearch instance? (y / N): y
Store username/password for secondary remote OpenSearch/Elasticsearch instance?? (y / N): y
OpenSearch/Elasticsearch username: remotedb
remotedb password:
remotedb password (again):
Require SSL certificate validation for OpenSearch/Elasticsearch communication? (Y / n): n
…
These files are created with permissions such that only the user account running Malcolm can access them:
$ ls -la .opensearch.*.curlrc
-rw------- 1 user user 36 Aug 22 14:17 .opensearch.primary.curlrc
-rw------- 1 user user 35 Aug 22 14:18 .opensearch.secondary.curlrc
One caveat with Malcolm using a remote OpenSearch as its primary document store is that the accounts used to access Malcolm’s web interfaces, particularly OpenSearch Dashboards, are passed directly through to OpenSearch itself. For this reason, both Malcolm and the remote primary OpenSearch instance must have the same account information. The easiest way to accomplish this is to use an Active Directory/LDAP server that both Malcolm and OpenSearch use as a common authentication backend.
See the OpenSearch documentation on access control or the Elasticsearch documentation on user authorization for more information.