Kamailio-基于Homer与heplify的SIP信令监控-2

heplify 是一个终端采集工具，主要将数据推送到 heplify-server 使用。heplify server是一个接受数据的服务，数据来自heplify，需要一个其启动的IP端口地址。homerr是一个api server，并提供UI 展示的能力。

c_zyer

982人浏览 · 2024-09-14 08:44:08

c_zyer · 2024-09-14 08:44:08 发布

接上篇，我们已经顺利地完成了服务的安装，下面就来看看如何配置并启动。

跟着我，你将学会：

下载并安装
- 踩坑：按照官方步骤来，可是网络条件不允许
配置并启动（本篇）
如何使用 Homer 查询会话信息

配置并启动

配置主要有几个点，这边主要是依赖 PostgreSQL 来进行数据存储，以下主要讲 PG 数据的初始化。

heplify

Portable and Lightweight HEP Capture Agent for HOMER, Captures packets from wire and sends them to Homer

heplify 是一个终端采集工具，主要将数据推送到 heplify-server 使用

可以手动启动，也可以将其定义为一个 systemctl 的service 进行运作。官方是没有要求的，我下文会以service的形式运行。但是测试的话，可以先手动启动。

heplify-server

HEP Capture Server for HOMER
server是一个接受数据的服务，数据来自heplify，需要一个其启动的IP端口地址。

数据主要服务于前端的Homer，因此这边连接数据库，也主要是关心homer的schema、homer配置所在表、homer数据所在表，额外需要一个地址、用户。

我们来看看默认的配置文件，默认配置文件位于/opt/heplify-server/heplify-server.toml：

HEPAddr               = "0.0.0.0:9060"
HEPTCPAddr            = ""
HEPTLSAddr            = "0.0.0.0:9060"
HEPWSAddr             = "0.0.0.0:3000"
ESAddr                = ""
ESDiscovery           = true
LokiURL               = ""
LokiBulk              = 200
LokiTimer             = 4
LokiBuffer            = 100000
LokiHEPFilter         = [1,5,100]
LokiAllowOutOfOrder   = false
ForceHEPPayload	      = []
PromAddr              = ""
PromTargetIP          = ""
PromTargetName        = ""
DBShema               = "homer7"
DBDriver              = "postgres"
DBAddr                = "localhost:5432"
DBUser                = "postgres"
DBPass                = ""
DBDataTable           = "homer_data"
DBConfTable           = "homer_config"
DBBulk                = 200
DBTimer               = 4
DBBuffer              = 400000
DBWorker              = 8
DBRotate              = true
DBPartLog             = "2h"
DBPartSip             = "1h"
DBPartQos             = "6h"
DBDropDays            = 14
DBDropDaysCall        = 0
DBDropDaysRegister    = 0
DBDropDaysDefault     = 0
DBDropOnStart         = false
DBUsageProtection     = true
DBUsageScheme         = "percentage"
DBPercentageUsage     = "80%"
DBMaxSize             = "30MB"
Dedup                 = false
DiscardMethod         = []
AlegIDs               = []
CustomHeader          = []
SIPHeader             = []
LogDbg                = ""
LogLvl                = "info"
LogStd                = false
LogSys                = false
Config                = "./heplify-server.toml"
ConfigHTTPAddr        = ""

# Examples:
# -------------------------------------
# ESAddr          = "http://127.0.0.1:9200"
# DBShema         = "homer7"
# DBDriver        = "postgres"
# LokiURL         = "http://localhost:3100/api/prom/push"
# LokiHEPFilter   = [1,5,100]
# PromAddr        = "0.0.0.0:8899"
# PromTargetIP    = "10.1.2.111,10.1.2.4,10.1.2.5,10.1.2.6,10.12.44.222"
# PromTargetName  = "sbc_access,sbc_core,kamailio,asterisk,pstn_gateway"
# AlegIDs         = ["X-CID","P-Charging-Vector,icid-value=\"?(.*?)(?:\"|;|$)","X-BroadWorks-Correlation-Info"]
# DiscardMethod   = ["OPTIONS","NOTIFY"]
# CustomHeader    = ["X-CustomerIP","X-Billing"]
# SIPHeader       = ["callid","callid_aleg","method","ruri_user","ruri_domain","from_user","from_domain","from_tag","to_user","to_domain","to_tag","via","contact_user"]
# LogDbg          = "hep,sql,loki"
# LogLvl          = "warning"
# ConfigHTTPAddr  = "0.0.0.0:9876"
# -------------------------------------
# To hot reload PromTargetIP and PromTargetName run:
# killall -HUP heplify-server

homer

Homer API Server and UI Webapplication

homer 的配置就是需要数据库的连接，不同的账号密码。兼容的组件和玩法也比较多，其他的配置项也是比较多的。

默认配置文件位于/usr/local/homer/etc/webapp_config.json:

{
    "database_data": {
        "LocalNode": {
            "help": "Settings for PGSQL Database (data)",
            "node": "LocalNode",
            "user": "homer_user",
            "pass": "homer_password",
            "name": "homer_data",
            "keepalive": true,
            "host": "127.0.0.1"
        }
    },
    "hep_relay": {
        "help": "UDP socket to send HEP data on",
        "host": "127.0.0.1",
        "port": 9060
    },
    "database_config": {
        "help": "Settings for PGSQL Database (settings)",
        "node": "LocalConfig",
        "user": "homer_user",
        "pass": "homer_password",
        "name": "homer_config",
        "keepalive": true,
        "host": "localhost"
    },
    "influxdb_config": {
        "enable": true,
        "help": "Settings for InfluxDB Database (optional)",
        "user": "influx_user",
        "pass": "influx_password",
        "name": "homer_config",
        "host": "http://127.0.0.1:8086",
        "database": "homer",
        "policy": "autogen"
    },
    "prometheus_config": {
        "enable": true,
        "help": "Settings for Prometheus Database (optional)",
        "user": "admin",
        "pass": "admin",
        "host": "http://127.0.0.1:9090",
        "api": "api/v1"
    },
    "swagger": {
        "enable": true,
        "api_json": "/usr/local/homer/etc/swagger.json",
        "api_host": "127.0.0.1:9080"
    },
    "loki_config": {
        "help": "Settings for LOKI Database (optional)",
        "user": "admin",
        "password": "admin",
        "param_query": "query_range",
        "regexp": false,
        "host": "http://127.0.0.1:3100",
        "template": "{job=\"heplify-server\"} |~ \"%callid%\"",
        "external_url": "",
        "api": "loki/api/v1",
        "enable": true
    },
    "grafana_config": {
        "help": "Settings for Grafana",
        "host": "http://127.0.0.1:3000",
        "path": "grafana",
        "token": "",
        "proxy_control": false,
        "proxy_check": "simple",
        "enable": true
    },
    "http_settings": {
        "help": "Settings for the HOMER Webapp Server. If you have gzip_static = false, please be sure that your dist directory has uncompressed .js files",
        "host": "0.0.0.0",
        "port": 9080,
        "root": "/usr/local/homer/dist",
        "gzip": true,
        "path": "/",
        "gzip_static": true,
        "debug": false
    },
    "transaction_settings": {
        "deduplicate": {
            "global": false
        }
    },
    "api_settings": {
        "enable_token_access": false,
        "add_captid_to_resolve": false
    },
    "https_settings": {
        "help": "SSL settings for homer-app",
        "enable": false,
        "host": "0.0.0.0",
        "port": 443,
        "min_tls_version": "1.2",
        "use_mozilla_profile": true,
        "cert": "/usr/local/homer/tls/cert.pem",
        "key": "/usr/local/homer/tls/key.pem"
    },
    "system_settings": {
        "help": "Settings for HOMER logs",
        "logpath": "/usr/local/homer/log",
        "logname": "homer-app.log",
        "_loglevels": "can be: fatal, error, warn, info, debug, trace",
        "loglevel": "error",
        "logstdout": false
    },
    "dashboard_settings": {
        "_comment": "Here you can define a customs dashboards home",
        "dashboard_home": "/usr/local/homer/etc/dashboard_home.json"
    },
    "auth_settings": {
        "_comment": "The type param can be internal, ldap, http_auth",
        "type": "internal",
        "_jwt_help": "Please generate own jwt secret using uuid or the flag generate-jwt-secret",
        "jwt_secret": "",
        "gravatar": false,
        "auth_token_header": "Auth-Token",
        "gravatar_url": "https://www.gravatar.com/avatar/%s.jpg",
        "token_expire": 1200,
        "user_groups": [
            "admin",
            "user",
            "support"
        ]
    },
    "ldap_config": {
        "base": "dc=example,dc=com",
        "host": "ldap.example.com",
        "port": 389,
        "usessl": false,
        "skiptls": true,
        "binddn": "uid=readonlysuer,ou=People,dc=example,dc=com",
        "bindpassword": "readonlypassword",
        "userfilter": "(uid=%s)",
        "groupfilter": "(memberUid=%s)",
        "group_attributes": [
            "cn",
            "memberOf",
            "GroupAttribute",
            "distinguishedName",
            "dn",
            "member"
        ],
        "admingroup": "admin",
        "adminmode": true,
        "usergroup": "HOMER_user",
        "usermode": true,
        "attributes": [
            "dn",
            "givenName",
            "sn",
            "mail",
            "uid"
        ],
        "skipverify": true,
        "anonymous": false,
        "userdn": "uid=%s,ou=People,dc=example,dc=com"
    },
    "http_auth": {
        "url": "http://localhost:1323",
        "skipverify": true
    },
    "oauth2": {
        "enable": false,
        "client_id": "1234565",
        "client_secret": "FAKE",
        "project_id": "Homer OAuth",
        "auth_uri": "https://accounts.google.com/o/oauth2/auth",
        "token_uri": "https://oauth2.googleapis.com/token",
        "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
        "redirect_uri": "http://localhost:80/api/v3/oauth2/auth",
        "service_redirect": "/api/v3/oauth2/redirect",
        "profile_url": "https://www.googleapis.com/oauth2/v1/userinfo",
        "provider_name": "google",
        "grant_type": "authorization_code",
        "response_type": "code",
        "auth_style": 1,
        "user_token": "RandomURLSafeStringWithAMinimumLengthOf43Characters",
        "scope": [
            "email",
            "openid",
            "profile"
        ],
        "gravatar": false,
        "gravatar_url": "https://www.gravatar.com/avatar/%s.jpg",
        "provider_image": ""
    },
    "decoder_shark": {
        "_comment": "Here you can do packet decoding using tshark application. Please define uid, gid if you run the app under root",
        "active": false,
        "bin": "/usr/bin/tshark",
        "protocols": [
            "1_call",
            "1_registration",
            "1_default"
        ]
    }
}

配置数据库专用用户

对于homer服务，创建数据库专用用户。

使用管理员账户连接上PG后，执行。

create user homer with password 'homerpass';

初始化homer数据

从上面的配置就可以看到，有一个配置数据库和一个数据数据库，下面就对其进行初始化，需要用到pg的root用户。

创建配置数据库

$ homer-app -create-config-db -database-root-user=root -database-host=xx.xx.xx.xx -database-root-password=pass -database-homer-user=homer

CONNECT to DB ROOT STRING: [host=192.168.5.109 user=root dbname=postgres sslmode=disable port=5432 password=root]


HOMER - create db [homer_config] with [name=homer]

(/app/migration/migration.go:89)
[2024-08-20 18:08:30]  [148.72ms]  CREATE DATABASE "homer_config" OWNER "homer"
[0 rows affected or returned ]

DONE

HOMER - show users
        Role name       |       Attributes
------------------------------------------------
        homer   |
        root    |       superuser, create database
        test    |

DONE

登录确认：

psql (9.2.24, server 12.19)
WARNING: psql version 9.2, server version 12.0.
         Some psql features might not work.
Type "help" for help.

postgres=# \l
                               List of databases
     Name     | Owner | Encoding |  Collate   |   Ctype    | Access privileges
--------------+-------+----------+------------+------------+-------------------
 homer_config | homer | UTF8     | en_US.utf8 | en_US.utf8 |
 postgres     | root  | UTF8     | en_US.utf8 | en_US.utf8 |
 root         | root  | UTF8     | en_US.utf8 | en_US.utf8 |
 template0    | root  | UTF8     | en_US.utf8 | en_US.utf8 | =c/root          +
              |       |          |            |            | root=CTc/root
 template1    | root  | UTF8     | en_US.utf8 | en_US.utf8 | =c/root          +
              |       |          |            |            | root=CTc/root
 test         | root  | UTF8     | en_US.utf8 | en_US.utf8 | =Tc/root         +
              |       |          |            |            | root=CTc/root    +
              |       |          |            |            | test=CTc/root
(6 rows)

创建数据数据库

$ homer-app -create-data-db -database-root-user=root -database-host=xx.xx.xx.xx -database-root-password=pass -database-homer-user=homer

HOMER - create db [homer_data] with [name=homer]

(/app/migration/migration.go:89)
[2024-08-20 18:10:25]  [115.71ms]  CREATE DATABASE "homer_data" OWNER "homer"
[0 rows affected or returned ]

DONE

HOMER - show users
        Role name       |       Attributes
------------------------------------------------
        homer   |
        root    |       superuser, create database
        test    |

DONE

登录确认：

psql (9.2.24, server 12.19)
WARNING: psql version 9.2, server version 12.0.
         Some psql features might not work.
Type "help" for help.

postgres=# \l
                               List of databases
     Name     | Owner | Encoding |  Collate   |   Ctype    | Access privileges
--------------+-------+----------+------------+------------+-------------------
 homer_config | homer | UTF8     | en_US.utf8 | en_US.utf8 |
 homer_data   | homer | UTF8     | en_US.utf8 | en_US.utf8 |
 postgres     | root  | UTF8     | en_US.utf8 | en_US.utf8 |
 root         | root  | UTF8     | en_US.utf8 | en_US.utf8 |
 template0    | root  | UTF8     | en_US.utf8 | en_US.utf8 | =c/root          +
              |       |          |            |            | root=CTc/root
 template1    | root  | UTF8     | en_US.utf8 | en_US.utf8 | =c/root          +
              |       |          |            |            | root=CTc/root
 test         | root  | UTF8     | en_US.utf8 | en_US.utf8 | =Tc/root         +
              |       |          |            |            | root=CTc/root    +
              |       |          |            |            | test=CTc/root
(7 rows)

创建数据表并修改权限

$ homer-app -create-table-db-config

HOMER - creating tables for the config DB [dbname=homer_config]
Successful ping: 192.168.5.109, Type: config, Node: localnode
DONE

$ homer-app -populate-table-db-config
Successful ping: 192.168.5.109, Type: config, Node: localnodeINFO[0000] SELECT * FROM "versions"                      module=gorm type=sql

HOMER - filling tables for the config DB [dbname=homer_config]
 reinstalling users
 reinstalling global_settings
 reinstalling auth_token
 reinstalling agent_location_session
 reinstalling hepsub_mapping_schema
 reinstalling user_settings
 reinstalling mapping_schema
 reinstalling versions

DONE

$ homer-app -upgrade-table-db-config

HOMER - upgrading tables for the config DB [dbname=homer_config]
Successful ping: 192.168.5.109, Type: config, Node: localnode
DONE

完成以上步骤后，对应配置文件内的连接用户、地址、数据库名都做好调整，就可以准备启动服务了。

启动服务

启动heplify-server

$ systemctl start heplify-server
$ systemctl status heplify-server
● heplify-server.service - HEP Server & Switch in Go
   Loaded: loaded (/usr/lib/systemd/system/heplify-server.service; enabled; vendor preset: disabled)
   Active: active (running) since 五 2024-09-13 10:55:19 CST; 1s ago
 Main PID: 17504 (heplify-server)
   CGroup: /system.slice/heplify-server.service
           └─17504 /usr/local/bin/heplify-server -config=/etc/heplify-server.toml

9月 13 10:55:19 192.168.5.167 systemd[1]: Started HEP Server & Switch in Go.

启动homer-app

$ systemctl start homer-app
$ systemctl status homer-app
● homer-app.service - Homer API Server and UI Webapplication
   Loaded: loaded (/usr/lib/systemd/system/homer-app.service; disabled; vendor preset: disabled)
   Active: active (running) since 五 2024-09-13 10:55:59 CST; 1s ago
 Main PID: 17560 (homer-app)
   CGroup: /system.slice/homer-app.service
           └─17560 /usr/local/bin/homer-app

9月 13 10:56:00 192.168.5.167 homer-app[17560]: ___ /  /::\
9月 13 10:56:00 192.168.5.167 homer-app[17560]: /__/\  /:/\:\     _____
9月 13 10:56:00 192.168.5.167 homer-app[17560]: \  \:\/:/__\/    |___  |
9月 13 10:56:00 192.168.5.167 homer-app[17560]: \  \::/            / /
9月 13 10:56:00 192.168.5.167 homer-app[17560]: \  \:\           / /
9月 13 10:56:00 192.168.5.167 homer-app[17560]: \  \:\         /_/
9月 13 10:56:00 192.168.5.167 homer-app[17560]: \__\/
9月 13 10:56:00 192.168.5.167 homer-app[17560]: 
9月 13 10:56:00 192.168.5.167 homer-app[17560]:  Version: homer-app 1.5.3. 
9月 13 10:56:00 192.168.5.167 homer-app[17560]: ⇨ http server started on [::]:9080
[root@192 ~]#

最终启动 heplify

/opt/heplify/heplify -i lo -hs ip:9060 -m SIP -dim REGISTER -pr 5060-5090

它的使用方法简单讲，就是基于什么网口、针对什么类型网络流量、是否需要过滤、端口范围是什么，抓到的数据推到哪里。

我上面的配置就是：针对lo网口上，端口5060-5090范围内的SIP消息，除却REGISTER消息，其他抓取的到后送到ip:9060地址。

看看它的详细用法：

$ /opt/heplify/heplify --help
Use heplify 1.66.7 like: /opt/heplify/heplify [option]
  -assembly_debug_log
    	If true, the github.com/google/gopacket/tcpassembly library will log verbose debugging information (at least one line per packet)
  -assembly_memuse_log
    	If true, the github.com/google/gopacket/tcpassembly library will log information regarding its memory use every once in a while.
  -b int
    	Interface buffersize (MB) (default 32)
  -bpf string
    	Custom BPF to capture packets
  -collectonlysip
    	collect only sip
  -d string
    	Enable certain debug selectors [defrag,layer,payload,rtp,rtcp,sdp]
  -dd
    	Deduplicate packets
  -di string
    	Discard uninteresting packets by any string
  -didip string
    	Discard uninteresting SIP packets by Destination IP(s)
  -diip string
    	Discard uninteresting SIP packets by Source or Destination IP(s)
  -dim string
    	Discard uninteresting SIP packets by Method [OPTIONS,NOTIFY]
  -disip string
    	Discard uninteresting SIP packets by Source IP(s)
  -e	Log to stderr and disable syslog/file output
  -eof-exit
    	Exit on EOF of ReadFile
  -erspan
    	erspan
  -fg uint
    	Fanout group ID for af_packet
  -fi string
    	Filter interesting packets by any string
  -fnum int
    	The total num of log files to keep (default 7)
  -fsize uint
    	The rotate size per log file based on byte (default 10485760)
  -fw int
    	Fanout worker count for af_packet (default 4)
  -hep-buffer-activate
    	enable buffer messages if connection to HEP server broken
  -hep-buffer-debug
    	enable debug buffer messages
  -hep-buffer-file string
    	filename and location for hep-buffer file (default "HEP-Buffer.dump")
  -hep-buffer-max-size string
    	max buffer size, can be B, KB, MB, GB, TB. By default - unlimited (default "0")
  -hi uint
    	HEP node ID (default 2002)
  -hin string
    	HEP collector address [udp:127.0.0.1:9093]
  -hn string
    	HEP node Name
  -hp string
    	HEP node PW
  -hs string
    	HEP server address (default "127.0.0.1:9060")
  -i string
    	Listen on interface (default "any")
  -keepalive uint
    	keep alive internal - 5 seconds by default. 0 - disable (default 5)
  -l string
    	Log level [debug, info, warning, error] (default "info")
  -lp int
    	Loop count over ReadFile. Use 0 to loop forever (default 1)
  -m string
    	Capture modes [SIP, SIPDNS, SIPLOG, SIPRTCP] (default "SIPRTCP")
  -n string
    	Log filename (default "heplify.log")
  -nt string
    	Network types are [udp, tcp, tls] (default "udp")
  -o	Read packet for packet
  -p string
    	Log filepath (default "./")
  -pr string
    	Portrange to capture SIP (default "5060-5090")
  -prometheus string
    	prometheus metrics - ip:port. By default all IPs (default ":8090")
  -protobuf
    	Use Protobuf on wire
  -rf string
    	Read pcap file
  -rs
    	Use packet timestamps with maximum pcap read speed
  -rt int
    	Pcap rotation time in minutes (default 60)
  -s int
    	Snaplength (default 8192)
  -script-file string
    	Script file to execute on each packet
  -script-hep-filter string
    	HEP filter for script, comma separated list of HEP types (default "1")
  -sipassembly
    	If true, sipassembly will be enabled
  -skipverify
    	skip certifcate validation
  -sl
    	Log to syslog
  -t string
    	Capture types are [pcap, af_packet] (default "af_packet")
  -tcpassembly
    	If true, tcpassembly will be enabled
  -tcpsendretries uint
    	Number of retries for sending before giving up and reconnecting
  -version
    	Show heplify version
  -vlan
    	vlan
  -wf string
    	Path to write pcap file
  -zf
    	Enable pcap compression

命令行直接启动后，输出的东西很少，主要就是产生流量，然后去homer的页面看了。

因为命令行启动仅仅是测试，因为终端肯定还是会关掉的，那么service模式怎么配置呢？

$ cat /etc/systemd/system/heplify.service
  [Unit]
  Description=Captures packets from wire and sends them to Homer
  After=network.target
  
  [Service]
  WorkingDirectory=/opt/heplify
  ExecStart=/opt/heplify/heplify -i lo -hs ip:9060 -m SIP -dim REGISTER -pr "5060-5090"
  ExecStop=/bin/kill ${MAINPID}
  Restart=on-failure
  RestartSec=10s
  Type=simple
  
  [Install]
  WantedBy=multi-user.target

其实就是熟悉的那条命令，只不过交由 systemctl 来管理。手动修改完指令后，reload一下，就可以尝试启动了。

$ systemctl start heplify
$ systemctl status heplify
● heplify.service - Captures packets from wire and sends them to Homer
   Loaded: loaded (/etc/systemd/system/heplify.service; enabled; vendor preset: disabled)
   Active: active (running) since 五 2024-09-13 10:53:05 CST; 2s ago
  Process: 22817 ExecStop=/bin/kill ${MAINPID} (code=exited, status=0/SUCCESS)
 Main PID: 17288 (heplify)
   CGroup: /system.slice/heplify.service
           └─17288 /opt/heplify/heplify -i lo -hs ip:9060 -m SIP -dim REGISTER -pr 5060-5090

9月 13 10:53:05 ip systemd[1]: Started Captures packets from wire and sends them to Homer.

看看启动后的web页面成效：

在这里插入图片描述

后续再介绍每一个页面的功能和使用。

sipcapture对这块功能版图还是铺的比较大，结合kamailio也做了sipcature的模块，辅助实现更为更多会话或事务级别的功能，此外还使用过siptrace去发送hep数据。

这里有几点思考：

对于heplify，这个外部抓网口数据的，如果一个服务器上很多服务，各种各样的网络包数据，使用这个工具抓取并过滤，不知道是否合适。但是它的好处就是对于Kamailio没有任何影响。
对于siptrace，开源技术公司出品，他其实内部就是指定一个heplify-server的地址，和heplify是一样的情况，但是这个抓取就是KM的脚本可自行控制的，追或者不追都可以，模式可选，也可写入到KM自己的数据库中。相比较，KM自身发起可控，但对KM的流程需要增加一步，高并发的场景不知道效果性能如何？
对于sipcapture，商业化公司出品，围绕其他公司相关产品。个人觉得这个模块是取代heplify-server的，新部署一个KM使用sipcapture模块，来接受siptrace过来的数据，然后直接写到数据库中，类似于 heplify-server收到数据写入数据库的动作。介绍较少，版图太大，没搭建起来尝试，有尝试过的小伙伴可以分享一下。详见官方文档说明。

在这里插入图片描述