2.0.0

HTTP数据

简介

service_http_server input插件可以接收来自unix socket、http/https、tcp的请求，并支持sls协议、otlp等多种协议。

版本

Stable

配置参数

参数类型是否必选说明

Type

String

是

插件类型，固定为service_http_server

Format

String

否

数据格式。

支持格式：sls、prometheus、influxdb、otlp_logv1、 otlp_metricv1、pyroscope、statsd

v2版本支持格式:raw、prometheus、otlp_logv1、otlp_metricv1、otlp_tracev1

说明：raw格式以原始请求字节流传输数据

Address

String

否

监听地址。

Path

String

否

接收端点, 如Format 为 otlp_logv1 时, 默认端点为/v1/logs 。

ReadTimeoutSec

String

否

读取超时时间。

默认取值为:10s。

ShutdownTimeoutSec

String

否

关闭超时时间。

默认取值为:5s。

MaxBodySize

String

否

最大传输 body 大小。

默认取值为:64k。

UnlinkUnixSock

String

否

启动前如果监听地址为unix socket，是否进行强制释放。

默认取值为:true。

FieldsExtend

Boolean

否

是否支持非integer以外的数据类型(如String)

目前仅针对有 String、Bool 等额外类型的 influxdb Format 有效

QueryParams

[]String

否

需要解析到Group.Metadata中的请求参数。

解析结果会以KeyValue放入Metadata。默认取值为[]，即不解析。

仅v2版本有效

QueryParamPrefix

String

否

解析请求参数时需要添加的key前缀，如_query_param_。

前缀会直接拼接在每个QueryParam前，无额外连接符，默认取值为空，即不增加前缀。

仅v2版本有效

HeaderParams

[]String

否

需要解析到Group.Metadata中的header参数。

解析结果会以KeyValue放入Metadata。默认取值为[]，即不解析。

仅v2版本有效

HeaderParamPrefix

String

否

解析Header参数时需要添加的key前缀，如_header_param_。

前缀会直接拼接在每个HeaderParam前，无额外连接符，默认取值为空，即不增加前缀。

仅v2版本有效

DisableUncompress

Boolean

否

禁用对于请求数据的解压缩, 默认取值为:false

目前仅针对Raw Format有效

仅v2版本有效

Tags

map[String]String

否

输出数据默认携带标签

仅v1版本有效

DumpData

Boolean

否

[开发使用] 将接收的请求存储于本地文件, 默认取值为:false

DumpDataKeepFiles

Int

否

[开发使用] Dump文件保留文件数目, 文件按小时滚动, 此参数默认值为5, 表示保留5小时Dump 参数

AllowUnsafeMode

Boolean

否

是否允许unsafe模式的Decode，启用该模式，Decoder将可能利用go unsafe技术来加速解码，目前仅当Format=prometheus时有效(注：暂不支持Exemplar、Histogram)

样例

接收 OTLP 日志

  • 采集配置

enable: true
inputs:
  - Type: service_http_server
    Format: "otlp_logv1"
    Address: "http://127.0.0.1:12345"
flushers:
  - Type: flusher_stdout
    OnlyStdout: true

  • 输入

使用 opentelemetry-java-sdk构造数据，基于ExampleConfiguration.java、OtlpExporterExample.java 进行如下代码改造。

ExampleConfiguration 进行 OTLP SDK 初始化。

  static OpenTelemetrySdk initHTTPOpenTelemetry() {
    // Include required service.name resource attribute on all spans and metrics
    Resource resource =
        Resource.getDefault()
            .merge(Resource.builder().put(SERVICE_NAME, "OtlpExporterExample").build());

    OpenTelemetrySdk openTelemetrySdk =
        OpenTelemetrySdk.builder()
            .setLogEmitterProvider(
                SdkLogEmitterProvider.builder()
                    .setResource(resource)
                    .addLogProcessor(SimpleLogProcessor
                        .create(OtlpHttpLogExporter
                                .builder()
                                .setEndpoint("http://127.0.0.1:12345/v1/logs")
                                .build()))
                    .build())
            .buildAndRegisterGlobal();

    Runtime.getRuntime()
        .addShutdownHook(new Thread(openTelemetrySdk.getSdkLogEmitterProvider()::shutdown));

    return openTelemetrySdk;
  }

OtlpExporterExample 构造数据。

  OpenTelemetrySdk openTelemetry = ExampleConfiguration.initHTTPOpenTelemetry();

  LogEmitter logger = openTelemetry.getSdkLogEmitterProvider().get("io.opentelemetry.example");
  logger
    .logRecordBuilder()
    .setBody("log body1")
    .setAllAttributes(
      Attributes.builder()
                  .put("k1", "v1")
                  .put("k2", "v2").build())
    .setSeverity(Severity.INFO)
    .setSeverityText("INFO")
    .setEpoch(Instant.now())
    .setContext(Context.current())
    .emit();

  • 输出

{
    "time_unix_nano": "1663913736115000000",
    "severity_number": "9",
    "severity_text": "INFO",
    "content": "log body1",
    "attributes": "{\"k1\":\"v1\",\"k2\":\"v2\"}",
    "resources": "{\"service.name\":\"OtlpExporterExample\",\"telemetry.sdk.language\":\"java\",\"telemetry.sdk.name\":\"opentelemetry\",\"telemetry.sdk.version\":\"1.18.0\"}",
    "__time__": "1663913736"
}

接收 OTLP Logs/Metrics/Traces (v2)

注意：目前v2 pipeline尚不支持otlp logs请求，会丢弃数据。

  • 采集配置


enable: true
version: v2
inputs:
  - Type: service_http_server
    Format: "otlp_logv1"
    Address: "http://127.0.0.1:12344"
  - Type: service_http_server
    Format: "otlp_metricv1"
    Address: "http://127.0.0.1:12345"
  - Type: service_http_server
    Format: "otlp_tracev1"
    Address: "http://127.0.0.1:12346"
flushers:
  - Type: flusher_stdout
    OnlyStdout: true

接收字节流数据

  • 采集配置

enable: true
version: v2
inputs:
  - Type: service_http_server
    Format: "raw"
    Address: "http://127.0.0.1:12345"
    QueryParams:
      - "QueryKey"
    QueryParamPrefix: "_param_prefix_"
    HeaderParams:
      - "HeaderKey"
    HeaderParamPrefix: "_header_prefix_"
flushers:
  - Type: flusher_stdout
    OnlyStdout: true
    Tags: true

  • 输入

手动构造请求

curl --location --request POST 'http://127.0.0.1:12345?QueryKey=queryValue' --header 'HeaderKey: headerValue' --header 'Content-Type: text/plain' --data 'test_measurement,host=server01,region=cn value=0.5'

  • 输出

[Event] event 1, metadata map[_header_prefix_HeaderKey:headerValue _param_prefix_QueryKey:queryValue], tags map[__hostname__:579ce1e01dea]

{
    "eventType":"byteArray",
    "name":"",
    "timestamp":0,
    "observedTimestamp":0,
    "tags":{
    },
    "byteArray":"test_measurement,host=server01,region=cn value=0.5"
}

接收Pyroscope Agent 数据

Agent协议是否兼容

pyroscopde/nodjs

pprof

是

pyroscopde/.net

pprof

是

pyroscopde/.net-new

pprof

是

pyroscopde/java

JFR

是

pyroscopde/go

pprof

是

pyroscopde/php

raw tire

是

pyroscopde/ebpf

raw tire

是

pyroscopde/rust

raw groups

是

pyroscopde/ruby

raw groups

是

pyroscopde/python

raw groups

是

  • 采集配置 使用v1 版本表述使用protocol.Log 传递数据

enable: true
version: v1
inputs:
  - Type: service_http_server
    Format: "pyroscope"
    Address: "http://:4040"
    Endpoint: "/ingest"
    Cluster: "sls-mall"
    TagsInGroup: false
flushers:
  - Type: flusher_stdout
    OnlyStdout: true
    Tags: true

  • 输入

  1. 进入Pyroscope Golang Example

  2. 编译 Pyroscope Golang Example

    go build -o main .

  3. 启动Pyroscope Golang Example

    ./main

  • 输出

2023-02-21 14:15:20 {"name":"runtime.malg /Users/evan/sdk/go1.19.4/src/runtime/proc.go","stack":"runtime.newproc1 /Users/evan/sdk/go1.19.4/src/runtime/proc.go\nruntime.newproc.func1 /Users/evan/sdk/go1.19.4/src/runtime/proc.go\nruntime.systemstack /Users/evan/sdk/go1.19.4/src/runtime/asm_amd64.s","stackID":"afb4871c4cb30267","language":"go","type":"profile_mem","dataType":"CallStack","durationNs":"0","profileID":"e3620581-930d-4c51-8e52-cb752e75f1be","labels":"{\"__name__\":\"simple.golang.app\",\"_sample_rate_\":\"100\",\"cluster\":\"s=s-mall\"}","units":"count","valueTypes":"alloc_objects","aggTypes":"sum","val":"1260.00","__time__":"1676960120"}
2023-02-21 14:15:20 {"name":"runtime.malg /Users/evan/sdk/go1.19.4/src/runtime/proc.go","stack":"runtime.newproc1 /Users/evan/sdk/go1.19.4/src/runtime/proc.go\nruntime.newproc.func1 /Users/evan/sdk/go1.19.4/src/runtime/proc.go\nruntime.systemstack /Users/evan/sdk/go1.19.4/src/runtime/asm_amd64.s","stackID":"afb4871c4cb30267","language":"go","type":"profile_mem","dataType":"CallStack","durationNs":"0","profileID":"e3620581-930d-4c51-8e52-cb752e75f1be","labels":"{\"__name__\":\"simple.golang.app\",\"_sample_rate_\":\"100\",\"cluster\":\"s=s-mall\"}","units":"bytes","valueTypes":"alloc_space","aggTypes":"sum","val":"524496.00","__time__":"1676960120"}
2023-02-21 14:15:20 {"name":"runtime.malg /Users/evan/sdk/go1.19.4/src/runtime/proc.go","stack":"runtime.newproc1 /Users/evan/sdk/go1.19.4/src/runtime/proc.go\nruntime.newproc.func1 /Users/evan/sdk/go1.19.4/src/runtime/proc.go\nruntime.systemstack /Users/evan/sdk/go1.19.4/src/runtime/asm_amd64.s","stackID":"afb4871c4cb30267","language":"go","type":"profile_mem","dataType":"CallStack","durationNs":"0","profileID":"e3620581-930d-4c51-8e52-cb752e75f1be","labels":"{\"__name__\":\"simple.golang.app\",\"_sample_rate_\":\"100\",\"cluster\":\"s=s-mall\"}","units":"count","valueTypes":"inuse_objects","aggTypes":"sum","val":"1260.00","__time__":"1676960120"}
2023-02-21 14:15:20 {"name":"compress/flate.(*compressor).init /Users/evan/sdk/go1.19.4/src/compress/flate/deflate.go","stack":"compress/flate.NewWriter /Users/evan/sdk/go1.19.4/src/compress/flate/deflate.go\ncompress/gzip.(*Writer).Write /Users/evan/sdk/go1.19.4/src/compress/gzip/gzip.go\nruntime/pprof.(*profileBuilder).flush /Users/evan/sdk/go1.19.4/src/runtime/pprof/proto.go\nruntime/pprof.(*profileBuilder).pbSample /Users/evan/sdk/go1.19.4/src/runtime/pprof/proto.go\nruntime/pprof.(*profileBuilder).build /Users/evan/sdk/go1.19.4/src/runtime/pprof/proto.go\nruntime/pprof.profileWriter /Users/evan/sdk/go1.19.4/src/runtime/pprof/pprof.go","stackID":"d4e9448662480cdb","language":"go","type":"profile_mem","dataType":"CallStack","durationNs":"0","profileID":"e3620581-930d-4c51-8e52-cb752e75f1be","labels":"{\"__name__\":\"simple.golang.app\",\"_sample_rate_\":\"100\",\"cluster\":\"s=s-mall\"}","units":"count","valueTypes":"alloc_objects","aggTypes":"sum","val":"177.00","__time__":"1676960120"}
PreviousGPU数据NextServiceInput示例插件

Last updated