Prometheus protocols

Exposing metrics

Note

If you are using ClickHouse Cloud, you can expose metrics to Prometheus using the Prometheus Integration.

ClickHouse can expose its own metrics for scraping from Prometheus:

<prometheus>
    <port>9363</port>
    <endpoint>/metrics</endpoint>
    <metrics>true</metrics>
    <asynchronous_metrics>true</asynchronous_metrics>
    <events>true</events>
    <errors>true</errors>
    <histograms>true</histograms>
    <dimensional_metrics>true</dimensional_metrics>
</prometheus>

Section `<prometheus.handlers>` can be used to make more extended handlers.
This section is similar to [<http_handlers>](/interfaces/http) but works for prometheus protocols:

```xml
<prometheus>
    <port>9363</port>
    <handlers>
        <my_rule_1>
            <url>/metrics</url>
            <handler>
                <type>expose_metrics</type>
                <metrics>true</metrics>
                <asynchronous_metrics>true</asynchronous_metrics>
                <events>true</events>
                <errors>true</errors>
                <histograms>true</histograms>
                <dimensional_metrics>true</dimensional_metrics>
            </handler>
        </my_rule_1>
    </handlers>
</prometheus>

Settings:

Name	Default	Description
port	none	Port for serving the exposing metrics protocol.
endpoint	/metrics	HTTP endpoint for scraping metrics by prometheus server. Starts with /. Should not be used with the <handlers> section.
url / headers / method	none	Filters used to find a matching handler for a request. Similar to the fields with the same names in the <http_handlers> section.
metrics	true	Expose metrics from the system.metrics table.
asynchronous_metrics	true	Expose current metrics values from the system.asynchronous_metrics table.
events	true	Expose metrics from the system.events table.
errors	true	Expose the number of errors by error codes occurred since the last server restart. This information could be obtained from the system.errors as well.
histograms	true	Expose histogram metrics from system.histogram_metrics
dimensional_metrics	true	Expose dimensional metrics from system.dimensional_metrics

Check (replace 127.0.0.1 with the IP addr or hostname of your ClickHouse server):

curl 127.0.0.1:9363/metrics

Remote-write protocol

ClickHouse supports the remote-write protocol. Data are received by this protocol and written to a TimeSeries table (which should be created beforehand).

<prometheus>
    <port>9363</port>
    <handlers>
        <my_rule_1>
            <url>/write</url>
            <handler>
                <type>remote_write</type>
                <database>db_name</database>
                <table>time_series_table</table>
            </handler>
        </my_rule_1>
    </handlers>
</prometheus>

Settings:

Name	Default	Description
port	none	Port for serving the remote-write protocol.
url / headers / method	none	Filters used to find a matching handler for a request. Similar to the fields with the same names in the <http_handlers> section.
table	none	The name of a TimeSeries table to write data received by the remote-write protocol. This name can optionally contain the name of a database too.
database	none	The name of a database where the table specified in the table setting is located if it's not specified in the table setting.

Remote-read protocol

ClickHouse supports the remote-read protocol. Data are read from a TimeSeries table and sent via this protocol.

<prometheus>
    <port>9363</port>
    <handlers>
        <my_rule_1>
            <url>/read</url>
            <handler>
                <type>remote_read</type>
                <database>db_name</database>
                <table>time_series_table</table>
            </handler>
        </my_rule_1>
    </handlers>
</prometheus>

Settings:

Name	Default	Description
port	none	Port for serving the remote-read protocol.
url / headers / method	none	Filters used to find a matching handler for a request. Similar to the fields with the same names in the <http_handlers> section.
table	none	The name of a TimeSeries table to read data to send by the remote-read protocol. This name can optionally contain the name of a database too.
database	none	The name of a database where the table specified in the table setting is located if it's not specified in the table setting.

Configuration for multiple protocols

Multiple protocols can be specified together in one place:

<prometheus>
    <port>9363</port>
    <handlers>
        <my_rule_1>
            <url>/metrics</url>
            <handler>
                <type>expose_metrics</type>
                <metrics>true</metrics>
                <asynchronous_metrics>true</asynchronous_metrics>
                <events>true</events>
                <errors>true</errors>
                <histograms>true</histograms>
                <dimensional_metrics>true</dimensional_metrics>
            </handler>
        </my_rule_1>
        <my_rule_2>
            <url>/write</url>
            <handler>
                <type>remote_write</type>
                <table>db_name.time_series_table</table>
            </handler>
        </my_rule_2>
        <my_rule_3>
            <url>/read</url>
            <handler>
                <type>remote_read</type>
                <table>db_name.time_series_table</table>
            </handler>
        </my_rule_3>
    </handlers>
</prometheus>