Data Lake - Custom Source monitor

The Custom Source option in Data Lake allows you to push data from your applications using APIs or SDKs. This enables seamless monitoring and analysis, giving you complete control over your custom metrics.

Get started

Follow the below steps to push data via Custom Source

1. Click Integrate Now next to Custom Source to begin.
2. In the Add Data Lake Monitor page, provide the following details:
   • Data Lake Name: Provide a unique name for your monitor.
   • Associate with Monitor Group(s): Choose a monitor group from the drop-down list to logically group your monitors. Refer to Monitor Groups for more details on creating groups.
   • Include Tags: Select how you want to filter the data pushed via SDK/API.
     • All Tags: Push all available data.
     • Starts With: Push data that starts with a specific term.
     • Contains: Push data containing a specific keyword.
     • Ends With: Push data ending with a specific term.
3. Click Save.
   • You will be redirected to the dashboard, where you can see the newly created monitor.
   • You are all set. You must now push data to the newly added monitor.

Import data to the created monitor

You can push data via SDK or API.

Push data via SDK

This option is used to send raw data (unaggregated data), and the aggregation will be done at our end.

• Java
• Node.js

// Create Metric provider using the following code snippet
MetricProvider provider = new MetricProvider("");
DataLakeProvider dataLake = provider.createDataLake(""); // The app key associated with the Data-lake monitor.
DataLakeResource resource = dataLake.createResource(new HashMap(){{
put("host", "192.168.10.254");
}});// Set of tags to associate with a list of metrics created using the appkey resource

// Push data using different methods like Count, Rate, Gauge, Histogram.
resource.gauge(
"ResponseTime", // The metric name (field name)
180, // The numerical value of the metric
"ms",// Unit of the metric (Default unit will be Count)
new HashMap(){{
	put("host", "192.168.10.254");
}}); // Set of tags to be associated with the metric

metricProvider.count(
    "<app key>", // The app key associated with the Data-lake monitor
    "db_write", // The metric name (field name)
    2.0, // Numerical value of the metric
    Collections.singletonMap("host", "192.168.10.254") // Set of tags to associate with the metric
);

metricProvider.rate(
 "<app key>", 
 "disk_free",
  16868264.0,
  Collections.singletonMap("host", "192.168.10.254")
);

metricProvider.gauge(
 "<app key>", 
 "disk_free",
  16868264.0,
  Collections.singletonMap("host", "192.168.10.254")
);

metricProvider.histogram(
 "<app key>", 
 "disk_free",
  16868264.0,
  Collections.singletonMap("host", "192.168.10.254")
);

npm install site24x7_datalake_sdk

Site24x7LicenseKey="your-license-key-here"
APP_PORT=3015

const metrics = require('site24x7_datalake_sdk');

const appKey = 'your-actual-app-key-here';

const dataLake = metrics.createDataLake(appKey);

const dataLakeResource = dataLake.createResource({
  resourcetag1: 'resourcetag1',
  resourcetag2: 'resourcetag2',
});

dataLakeResource.count({
  name: "event_occurrences",
  value: 10,
  unit: "events",
  tags: {
    source: "api",
    type: "user_action"
  }
});

dataLakeResource.gauge({
  name: "memory_usage",
  value: 75,
  unit: "MB",
  tags: {
    server: "web-server-1",
    environment: "production"
  }
});

dataLakeResource.rate({
  name: "requests_per_second",
  value: 250,
  unit: "requests",
  tags: {
    source: "load_balancer",
    region: "us-east"
  }
});

dataLakeResource.histogram({
  name: "response_times",
  value: 120,
  unit: "ms",
  tags: {
    endpoint: "/api/users",
    method: "GET"
  }
});

Push data via API

This option is used to push the aggregated data.

1. Go to the data lake monitor listed and click on it.
2. Hover over the hamburger icon () next to the monitor's name.
3. Select Show Post URL from the drop-down menu.
   
   
4. Follow the on-screen instructions.
   
   Note

   Sample Payload:
   

   [
      {
         "metrics":[
            {
               "unit":"ms",
               "datavalues":[
                  {
                     "p0":0,
                     "p99":16,
                     "p100":17,
                     "time_stamp":"1739341985265",
                     "count":10,
                     "p50":12,
                     "value":100,
                     "p95":15,
                     "tags":{
                        "metric_tag_2":"metricTag2Data",
                        "metric_tag_1":"metricTag1Data"
                     }
                  }
               ],
               "name":"MetricName",
               "type":"rate",
               "description ":"Sample Description"
            }
         ],
         "tags":{
            "group_tag_1":"groupTag1data",
            "group_tag_2":"groupTag2data"
         }
      }
   ]

   Where,

   • value: Processed value of the metric.
   • count: Number of metrics recorded within the interval used for value calculation. If not provided, the default value is 1.

Supported units

Related articles

Add a Data Lake monitor

Add a REST API Metric monitor

Performance metrics of Data Lake

Data Lake dashboard

Add a Custom Attribute monitor