Collectors
Samurai Collectors are used to receive and transport telemetry from your security controls, network devices or cloud services to the Samurai platform.
There are two types of collectors:
1. Cloud Collector
- Deployed within the Samurai platform and is used to gather telemetry from cloud services and/or security controls. Various use cases exist with differing requirements based on the Product/Service you are integrating with Samurai:
- In some cases you simply need to complete the relevant integration and the cloud collector is automatically used.
- When we gather telemetry from public cloud storage (specifically Microsoft Azure storage accounts and Amazon Web Services (AWS) S3 buckets) you must first deploy a cloud collector within the Samurai platform that is used to monitor your cloud storage for updated telemetry files.
- When we ingest telemetry using Splunk HTTP Event Collection (HEC) you must also first deploy a cloud collector within the Samurai platform that is used to receieve telemetry data.
Requirements are clearly stated in the relevant Product Integration Guides.
2. Local Collector
- Deployed within your environment and is used to gather telemetry from your security controls and network devices. We have packaged the local collector to support multiple formats and envionments.
What type of Collector do you require?
This is dependent on the products you want to integrate with Samurai:
- For products deployed in your internal network, a Local Collector will be required to gather (pull & push) telemetry data and securely transfer it to the Samurai platform.
- For cloud based products providing API endpoints, a Cloud Collector will be used to pull the telemetry data and securely transfer it to the Samurai platform.
- For cloud based products utilizing streaming of telemetry data to cloud storage, a Cloud Collector is also required to retrieve the telemetry data and securely transfer it to the Samurai platform.
- For products that leverage streaming of telemetry via Splunk HTTP Event Collection (HEC), a Cloud Collector is required to receieve telemetry data to the Samurai platform.
Next steps:
- Review our Supported Integrations and associated Integration Guides to determine the collector type(s) required. Within each Integration Guide there is a table denoting use of a Local or Cloud Collector, alternatively this is displayed in the Samurai MDR portal when working through an integration.
- You may also choose to jump directly to the Samurai MDR portal and review integrations
- If you have determined you require a local collector then click on Samurai Local Collector and follow the steps to create, configure and install.
- If you have determined you require a cloud collector then click on Samurai Cloud Collector and follow the steps to create and configure.
1 - Samurai Local Collector
If you have determined that you require a local collector then follow the steps below to learn what you need to get started, create, configure and download a local collector from the Samurai MDR portal and ensure it is working as expected.
- Take a moment to understand what you need to get started
- Create, configure and download a Collector
- Install a Collector
- Validate Collector Status
- Collector Status Notifications
- What’s next?
- Deleting a Collector
What you need to get started
Access to the Samurai MDR portal and your specific tenant.
A hypervisor to run the virtual machine, for example VMware vSphere, Microsoft Hyper-V, Amazon EC2 or Azure Virtual Machine
Ensure to make any necessary updates to comply with the collectors connectivity requirements.
A static IP address for the collector and DNS server IP addresses unless you decide to use DHCP.
Access to your products to make necessary changes outlined within the relevant integration guide.
Minimum Virtual Machine Requirements
The following machine requirements will support up to 15K events per second (EPS) peak, 10K EPS sustained over a 24hr period, approx 800GB per day.
| CPU | 2 vCPU |
|---|
| Disk | 500GB disk |
| Memory | 4 GB |
Connectivity required for the Collector
The collector requires connectivity to resources outlined within the table below, you may need to update your security controls e.g firewall to allow this connectivity.
| Function | Protocol | Port | Source | Destination | Details |
|---|
| Enrolment, Telemetry | TCP | 443 | Collector | *.*.security.ntt
nttsecurity.io
.nttsecurity.io
.*.nttsecurity.io
samurai-xdr-prod-westeurope-xgliuoit.azure-api.net | All regular backend communication, telemetry |
| Remote Management | TCP | 443 | Collector | ra.cto.nttsecurity.io
deb.releases.teleport.dev
apt.releases.teleport.dev | Used for remote administration of collector (this is not mandatory and used when troubleshooting) |
| NTP | UDP | 123 | Collector | Client infrastructure (NTP server(s)) if configured in Samurai app
OR
0.ubuntu.pool.ntp.org
1.ubuntu.pool.ntp.org
2.ubuntu.pool.ntp.org
3.ubuntu.pool.ntp.org | Time synchronization |
| DNS | UDP | 53 | Collector | Client infrastructure (DNS server(s)) or external DNS servers (based on your collector configuration) | Domain name resolution |
| Ubuntu updates | TCP | 80, 443 | Collector | *.ubuntu.com
api.snapcraft.io | Ubuntu software repository |
| Container Management | TCP | 443 | Collector | docker.com
*.docker.com (private container registry)
docker.io (private container registry)
*.docker.io (private container registry) | Private container registry |
| Amazon Cloud dependencies | TCP | 443 | Collector | *.cloudfront.net | Amazon CDN used by Collector API |
| Log storage | TCP | 443 | Collector | *.s3.*.amazonaws.com | Amazon Cloud storage (this is not mandatory and used when troubleshooting) |
| Telemetry data | (based on product - see Integration guide) | Client Product | Collector | Frequent data transfer (based on product) | |
From your Samurai MDR portal tenant, click Telemetry and select Collectors from the main menu
Select Create Collector
Select Local collector
Complete the fields as required.
| Collector name | A nickname for the collector |
|---|
| Description (Optional) | A description of your collector, this could be the property name where installed |
| Location (Optional) | Useful if you have collectors in multiple locations |
| Hostname | A hostname for your collector |
| Proxy Server IP (Optional) | Optional HTTP proxy IP address |
| NTP Servers (Optional) | Input your own NTP server IP addresses |
| DHCP or Static | Determine whether the collector will use DHCP or specify your own static IP address and network information |
Select Create Collector once you have completed all relevant fields
Select the Collector you created by clicking the Name used in Step 2
Select Download
- Download the iso configuration file and also the relevant file needed for your hypervisor.
If you are creating multiple collectors, you only need to download the ova file once and can use it multiple times, the important file per collector is the configuration file (iso).*
Install a Collector
Based on your hypervisor follow the relevant section:
VMware vSphere
Follow the documentation from VMware:
- When asked to provide a virtual machine name, we suggest samurai-nttsh-collector
- Be sure to select the .ova file you downloaded when asked for the file to deploy your virtual machine from.
Once complete follow the VMware article to configure a datastore ISO file
- Be sure to select the .iso file you downloaded when asked to select file
The VM is now ready to be powered on.
The .iso file must be mounted at first boot to configure the Collector. Once you have
validated the Collector status is Healthy in the Samurai MDR portal you must ensure the .iso is dismounted.*
Microsoft Hyper-V
Follow the documentation from Microsoft:
- When asked to provide a virtual machine name, we suggest samurai-nttsh-collector
- Use the Virtual Machine Requirements when configuring memory and network
- When asked to Connect Virtual Hard Disk ensure to use the .vhdx file you previously downloaded
- For Installation Options ensure you use the .iso file you previously downloaded
Once you have completed setup of your Collector you should ensure it is running and validate the status within the Samurai MDR portal, upon initial setup this can take a little while.
Amazon EC2
Prerequisitve steps:
- Ensure you have the AWS cloud-init.yaml file you downloaded from Create, Configure and Download a Collector.. This file will be used later during EC2 instance deployment.
Follow the vendor documentation from Amazon to launch a EC2 instance:
Perform the following adjustments to the vendor documentation when launching the instance:
- During step 4.a, select Ubuntu as AMI.
- During step 4.b*,* select the latest Ubuntu AMI
- During step 5*,* select a suitable Instance Type based on estimated performance requirements while fulfilling the Minimum Virtual Machine Requirements.
- During step 6 & 7, Set Key pair & Network Settings as per your AWS policies. Ensuring the the Network settings still fulfills the Connectivity required for the Collector.
- Before step 8, modify the Configure storage section with the following settings:
- Adjust the Root Volume to be at least 64 GiB.
- Add a secondary volume with at least 500 GiB according to the Minimum Virtual Machine Requirements.
Secondary disk volume will be used for spooling, size it according to estimated log volume and max downtime.
- Before step 8, expand the section Advanced details and paste in the content of the cloud-init.yaml file into the User data section. Ensure that the check box User data has already been base64 encoded is not enabled.
- Proceed with step 8 and finish the rest of the installation as per the vendor documentation.
Azure Virtual Machine
Prerequisite steps:
- Ensure you have the Azure cloud-init.yaml file you downloaded from Create, Configure and Download a Collector.. This file will be used later during the Virtual Machine instance deployment.
Follow the vendor documentation from Microsoft to launch a Virtual Machine instance:
Perform the following adjustments to the vendor documentation when launching the instance:
- Under the Basic tab, select Ubuntu Server 22.04 LTS as image
- Under the Basic tab, select a suitable Size based on estimated performance requirements while fulfilling the Minimum Virtual Machine Requirements.
- Under the Disk tab, add one data disk with at least 500 GiB according to the Minimum Virtual Machine Requirements.
Data disk volume will be used for spooling, size it according to estimated log volume and max downtime.
- Under the Advanced tab, paste the contents of cloud-init.yaml in the Custom datafield.
All other settings such authentication, network configuration and monitoring should be configured according to company policy and best practices.*
Validate Collector Status
Click Telemetry and select Collectors from the main menu
Select the relevant Collector from the presented list
View Status
| Status | Description |
|---|
| Offline | Collector created but not online |
| Unavailable | Collector has been online but no longer available |
| Healthy | Collector deployed and deployed add on components (including) Integrations and/or Evidence Fetchers) |
| Not-Healthy | Component(s) deployed on the Collector not healthy |
| Provisioning | Collector is in setup |
After you provision a Collector VM and start it, it will go through a process of installing updates and modules specified in the configuration ISO file which you downloaded. The time taken for this process is dependent on factors like the speed of the hardware you are running the Collector on and connectivity to the repositories that it downloads updates from. In some cases this process can take around 30 minutes.
The Collector may show as “Offline” during the initial provisioning steps. This is not any cause for alarm.
If you have any problems, please submit a ticket via the Samurai MDR portal.
Collector Status Notifications
Samurai will send email notifications to registered application users should your Local Collector status change from Healthy to Not-Healthy or Unavailable. Once any issues have been resolved, you will also be notified again when a Healthy status is reached.
If your Local Collector be restarted, during final startup you may notice the Status change from Healthy to Not-Healthy, this is not cause for alarm as this typically occurs for a short period of time as processes restart. Once complete your Local Collector status will be displayed as Healthy.
What’s next?
You should now have a collector running within your environment!
The next step is to start configuring integrations which will allow the Samurai platform to start receiving your telemetry data.
Select Integrations Overview for more information on integrations and where to start.
If you require high availability for your collector, this can be achieved using the capabilities of your virtualization platform.
Deleting a Collector
If you delete a local collector it cannot be reversed! In addition, all of your integrations related to the local collector will also be deleted!
If you need to delete a local collector you can do so by following the steps below:
- From your Samurai MDR portal click Telemetry and select Collectors
- Select the relevant collector from your list
- On the right hand side of the relevant collector, click on
(more options) and select Delete Collector - The following warning will appear: ‘Warning: This is a destructive action and cannot be reversed.’. To ensure you intended to delete the collector you will need to type DELETE in the field and select Delete Collector
Replacing a Collector
If for some reason a Local Collector VM is lost due to corruption or damage, such as in the case of a major disk storage failure, you may need to replace your Collector. If this happens, you will need to delete the old Collector in the Samurai MDR portal, discard your old Collector VM image and then create a new Collector using the process described to Install a Collector.
- If you need to replace a Collector VM, you cannot re-download the installer ISO for an existing Collector and redeploy it. You must delete the old Collector and replace it with a new one.
- You can re-use the same IP address as your old Collector. This allows you to replace a Collector without re-configuring any log sources which were sending logs to the old Collector.
- When replacing a Collector, any Integrations which were automatically detected and attached to the original Collector will be automatically detected and attached to the new Collector.
- Once you have created the new Collector, you will need to add any Integrations which you were previously using and which you had to previously manually add to the old Collector.
2 - Samurai Cloud Collector
Deployed within the Samurai platform, Cloud Collectors are used to:
- Pull telemetry data via an API and securely transfer it to the Samurai platform.
- Retrieve telemetry data from public cloud storage (Microsoft Azure storage accounts and Amazon Simple Storage Service - S3). The collector monitors for new or updated files in cloud storage and pulls the data into the Samurai platform for ingestion.
- Receieve telemetry data for ingestion to the Samurai platform (Splunk HTTP Event Collection - HEC).
The need for a Cloud Collector is based on the specific product being integrated with the Samurai platform. This will be clearly indicated within the Product Integration Guide.
- For integrations where we leverage an API, you can simply follow the integration guide as a cloud collector will already be available.
- For integrations where we leverage collection of telemetry data from public cloud storage or receipt via Splunk HTTP Event Collector (HEC) there are steps you will need to follow as outlined in the section below Create Cloud Collector.
Create Cloud Collector
For public cloud storage we recommend a minimum cloud storage retention period of 7 days (our template default), however you can update as necessary if a longer retention period is required.
From the Samurai MDR portal, click Telemetry and select Collectors from the main menu
Select Create Collector
Select Cloud collector
Complete the fields as required.
| Field | Description |
|---|
| Collector name | A name for the collector |
| Description (Optional) | A description of your collector |
| Provider | Select the correct Provider |
Select Create Collector
Follow the relevant section below based on your provider.
Microsoft Azure
- Click Deploy to Azure and you will be redirected to the Microsoft Azure login.
The Deploy to Azure button will only be displayed once after selecting Create Collector, therefore be sure to click the button before closing the dialog window.
- An Azure Resource Manager (ARM) template will be launched, follow the steps and complete the necessary fields within the template:
Project Details
| Field | Description |
|---|
| Subscription | Select your Azure subscription |
| Resource Group | Create or select your Resource Group |
Instance Details
| Field | Description |
|---|
| Region | Select the Azure region to deploy the Collector into |
| Collector Name | (this is auto populated from the Samurai MDR portal Collector name you defined) |
| Collector Id | (this is auto populated from Samurai) |
| Passkey | (this is auto populated from Samurai) |
Select Next
Select Review and Create
Upon creation your Collector status will be updated to Healthy.
You can now refer to the relevant Product Integration Guides.
Amazon Web Services (AWS)
- Click Launch Stack and you will be redirected to the AWS login.
The Launch Stack button will only be displayed once after selecting Create Collector, therefore be sure to click the button before closing the dialog window.
Login to your AWS account with administrative permissions.
The Samurai cloud formation template will be displayed.
If you have an existing S3 Bucket enter the name within the Parameters section under Enable Samurai ingestion on existing S3 bucket. If you have no existing S3 bucket, leave this field blank and a new S3 bucket will be created.
If you are integrating Cisco Umbrella, be sure to update Cisco Umbrella under the Parameters section to Yes.
Click Create Stack.
Upon creation of the stack your Collector status will be updated to Healthy.
You can now refer to the relevant Product Integration Guides.
Splunk HTTP Event Collector
- If you selected this option you will be presented with the following information:
Copy these entries as you will need them when completing your integration.
The API information is only displayed once after clicking Create Collector therefore be sure to save the information before clicking close.
Select Close.
You can now refer to the relevant Product Integration Guides.
Validate Collector Status
Select Telemetry and Collectors from the main menu
Select the relevant Collector from the presented list
View Status
| Status | Description |
|---|
| Offline | Collector created but offline |
| Not available | Collector has been online but no longer available |
| Healthy | Collector deployed and healthy |
| Not-Healthy | Collector not healthy |
| Provisioning | Collector is being setup / provisioning |
What’s next?
You should now have a collector running.
The next step is to start configuring integrations which will allow the Samurai platform to collect your telemetry data.
Select Integrations Overview for more information on integrations and where to start.
Deleting a Collector
If you delete a Cloud collector it cannot be reversed! In addition, all of your integrations related to the collector will also be deleted!
If you need to delete a Cloud collector you can do so by following the steps below:
- From your Samurai MDR portal click Telemetry and select Collectors from the main menu
- Select the relevant collector from your list
- On the right hand side of the relevant collector, click on
(more options) and select Delete Collector - The following warning will appear: ‘Warning: This is a destructive action and cannot be reversed.’. To ensure you intended to delete the collector you will need to type DELETE in the window and select Delete Collector