DataDog · bonnefoa · Jul 5, 2024 · Jul 5, 2024 · Jul 5, 2024 · Jul 5, 2024
diff --git a/README.md b/README.md
@@ -4,11 +4,19 @@
 
 pg_tracing is a PostgreSQL extension allows to generate server-side spans for distributed tracing.
 
-When pg_tracing is active, it generates spans on sampled queries. To access these spans, the extension provides two views: `pg_tracing_consume_spans` and `pg_tracing_peek_spans`. The utility functions `pg_tracing_reset` and `pg_tracing_info` provide ways to read and reset extension's statistics. These are not available globally but can be enabled for a specific database with `CREATE EXTENSION pg_tracing`.
+When pg_tracing is active, it generates spans on sampled queries. To access these spans, the extension provides two ways:
+- `pg_tracing_consume_spans` and `pg_tracing_peek_spans` views output spans as a set of records
+- `pg_tracing_json_spans` function outputs spans as a OTLP json
 
-Trace propagation currently relies on [SQLCommenter](https://google.github.io/sqlcommenter/). More mechanisms will be added in the future.
+The utility functions `pg_tracing_reset` and `pg_tracing_info` provide ways to read and reset extension's statistics. These are not available globally but can be enabled for a specific database with `CREATE EXTENSION pg_tracing`.
 
-> [!WARNING]  
+There are currently two mechanisms to propagate trace context:
+- As a SQL comment using [SQLCommenter](https://google.github.io/sqlcommenter/)
+- As a GUC parameter `pg_tracing.trace_context`
+
+See [Trace Propagation](#trace-propagation) for more details.
+
+> [!WARNING]
 > This extension is still in early development and may be unstable.
 
 ## PostgreSQL Version Compatibility
@@ -25,7 +33,8 @@ pg_tracing generates spans for the following events:
 - Execution Plan: A span is created for each node of the execution plan (SeqScan, NestedLoop, HashJoin...)
 - Nested queries: Statements invoked within another statement (like a function)
 - Triggers: Statements executed through BEFORE and AFTER trigger are tracked
-- Parallel Workers: Processes created to handle queries like Parallel SeqScans are tracked 
+- Parallel Workers: Processes created to handle queries like Parallel SeqScans are tracked
+- Transaction Commit: Time spent fsync changes on the WAL
 
 ## Documentation
 
@@ -48,11 +57,14 @@ To compile and install the extension, run:
 git clone https://github.com/DataDog/pg_tracing.git
 cd pg_tracing
 make install
+# To compile and install with debug symbols:
+PG_CFLAGS="-g" make install
 ```
 
 ## Setup
 
-The extension must be loaded by adding pg_tracing to the [shared_preload_libraries](https://www.postgresql.org/docs/current/runtime-config-client.html#GUC-SHARED-PRELOAD-LIBRARIES) in `postgresql.conf`. A server restart is needed to add or remove the extension.
+The extension must be loaded by adding pg_tracing to the [shared_preload_libraries](https://www.postgresql.org/docs/current/runtime-config-client.html#GUC-SHARED-PRELOAD-LIBRARIES) in `postgresql.conf`.
+A server restart is needed to add or remove the extension.
 
 ```
 # postgresql.conf
@@ -61,13 +73,17 @@ shared_preload_libraries = 'pg_tracing'
 compute_query_id = on
 pg_tracing.max_span = 10000
 pg_tracing.track = all
+
+# Send spans every 2 seconds to an otel collector
+pg_tracing.otel_endpoint = http://127.0.0.1:4318/v1/traces
+pg_tracing.otel_naptime = 2000
 ```
 
 The extension requires additional shared memory proportional to `pg_tracing.max_span`. Note that this memory is consumed whenever the extension is loaded, even if no spans are generated.
 
-When `pg_tracing` is active, it generates spans on sampled queries. To access these spans, the extension provides two views: `pg_tracing_consume_spans` and `pg_tracing_peek_spans`. The utility functions `pg_tracing_reset` and `pg_tracing_info` provide ways to read and reset extension's statistics. These are not available globally but can be enabled for a specific database with `CREATE EXTENSION pg_tracing`.
+## Trace Propagation
 
-## Usage
+### SQLCommenter
 
 Trace context can be propagated through [SQLCommenter](https://google.github.io/sqlcommenter/). By default, all queries with a SQLCommenter with a sampled flag enabled will generate spans.
 
@@ -84,6 +100,32 @@ select trace_id, parent_id, span_id, span_start, span_end, span_type, span_opera
  00000000000000000000000000000123 | 4268a4281c5316dd | f5994f9159d8e80d | 2024-03-19 13:46:43.980081+00 | 2024-03-19 13:46:43.980111+00 | Executor     | ExecutorRun
 ```
 
+### `trace_context` GUC
+
+The GUC variable `pg_tracing.trace_context` can also be used to propagate trace context.
+
+```sql
+BEGIN;
+SET LOCAL pg_tracing.trace_context='traceparent=''00-00000000000000000000000000000005-0000000000000005-01''';
+UPDATE pgbench_accounts SET abalance=1 where aid=1;
+COMMIT;
+
+-- Check generated span
+select trace_id, span_start, span_end, span_type, span_operation from pg_tracing_consume_spans order by span_start;
+             trace_id             |          span_start           |           span_end            |     span_type     |                      span_operation
+----------------------------------+-------------------------------+-------------------------------+-------------------+-----------------------------------------------------------
+ 00000000000000000000000000000005 | 2024-07-05 14:24:55.305234+00 | 2024-07-05 14:24:55.305988+00 | Update query      | UPDATE pgbench_accounts SET abalance=$1 where aid=$2;
+ 00000000000000000000000000000005 | 2024-07-05 14:24:55.305266+00 | 2024-07-05 14:24:55.30552+00  | Planner           | Planner
+ 00000000000000000000000000000005 | 2024-07-05 14:24:55.305586+00 | 2024-07-05 14:24:55.305906+00 | ExecutorRun       | ExecutorRun
+ 00000000000000000000000000000005 | 2024-07-05 14:24:55.305591+00 | 2024-07-05 14:24:55.305903+00 | Update            | Update on pgbench_accounts
+ 00000000000000000000000000000005 | 2024-07-05 14:24:55.305593+00 | 2024-07-05 14:24:55.305806+00 | IndexScan         | IndexScan using pgbench_accounts_pkey on pgbench_accounts
+ 00000000000000000000000000000005 | 2024-07-05 14:24:55.649757+00 | 2024-07-05 14:24:55.649792+00 | Utility query     | COMMIT;
+ 00000000000000000000000000000005 | 2024-07-05 14:24:55.649787+00 | 2024-07-05 14:24:55.649792+00 | ProcessUtility    | ProcessUtility
+ 00000000000000000000000000000005 | 2024-07-05 14:24:55.649816+00 | 2024-07-05 14:24:55.650613+00 | TransactionCommit | TransactionCommit
+```
+
+### Standalone Sampling
+
 Queries can also be sampled randomly through the `pg_tracing.sample_rate` parameter. Setting this to 1 will trace all queries.
 
 ```sql
@@ -102,6 +144,14 @@ select trace_id, parent_id, span_id, span_start, span_end, span_type, span_opera
  458fbefd7034e670eb3d9c930862c378 | bdecb6e35d429f3d | 8805f7749249536b | 2024-01-10 09:54:16.321485+00 | 2024-01-10 09:54:16.321529+00 | Executor     | ExecutorRun
 ```
 
-## Authors
+## Sending Spans
+
+Spans can be automatically sent to an otel collector by setting `pg_tracing.otel_endpoint` parameter:
+
+```
+# postgresql.conf
+pg_tracing.otel_endpoint = http://127.0.0.1:4318/v1/traces
+pg_tracing.otel_naptime = 2000
+```
 
-* [Anthonin Bonnefoy](https://github.com/bonnefoa)
+If an otel endpoint is defined, a background worker will be started and will send spans every $naptime using OTLP HTTP/JSON.
diff --git a/doc/pg_tracing.md b/doc/pg_tracing.md
@@ -10,12 +10,18 @@ pg_tracing extension allows to generate spans from with a PostgreSQL instance, p
 
 ## Tracecontext Propagation
 
-A query with SQLcommenter can propagate a trace context to the database. It will have the following format: 
+SQLcommenter or `pg_tracing.trace_context` can be used to propagate a trace context to the database. It will have the following format: 
 ```sql
 /*dddbs='postgres.db',traceparent='00-00000000000000000000000000000009-0000000000000005-01'*/ select 1;
 
 -- SQLCommenter may be added to the end
 select 1 /*dddbs='postgres.db',traceparent='00-00000000000000000000000000000009-0000000000000005-01'*/
+
+-- Propagation through trace_context GUC
+BEGIN;
+SET LOCAL pg_tracing.trace_context='traceparent=''00-00000000000000000000000000000005-0000000000000005-01''';
+UPDATE pgbench_accounts SET abalance=1 where aid=1;
+COMMIT;
 ```
 
 The traceparent fields are detailed in [w3c's trace-context](https://www.w3.org/TR/trace-context/#traceparent-header-field-values)
@@ -26,13 +32,13 @@ The traceparent fields are detailed in [w3c's trace-context](https://www.w3.org/
 01: trace flags (01 == sampled)
 ```
 
-When a query with SQLCommenter is detected, the trace context is extracted and used by pg_tracing.
+When a query with a trace context is detected, it is extracted and used by pg_tracing.
 
 ## Sampling
 
 Spans will only be generated for sampled queries. A query is sampled if:
-- It has a trace context propagated through SQLCommenter with the sampled flag enabled and it passes the `pg_tracing.caller_sample_rate`
-- It has no SQLCommenter but the query randomly passes the global `pg_tracing.sample_rate`
+- It has a trace context with the sampled flag enabled and it passes the `pg_tracing.caller_sample_rate`
+- It has no trace context but the query randomly passes the global `pg_tracing.sample_rate`
 
 With the default values `pg_tracing.caller_sample_rate = 1` and `pg_tracing.sample_rate = 0`, only queries with a trace context and a sampled flag `sampled := 01` will be sampled, effectively offloading sampling decision to the callers.
 
@@ -84,7 +90,7 @@ The spans generated by the module are made available via a view named `pg_tracin
 | `jit_inlining_time` | double precision | Total time spent by the node on inlining functions, in milliseconds |
 | `jit_optimization_time` | double precision | Total time spent by the node on optimizing, in milliseconds |
 | `startup` | bigint | Time to the first tuple in nanoseconds |
-| `parameters` | text | Value of the query's parameters |
+| `parameters` | text[] | Value of the query's parameters |
 | `deparse_info` | text | Information extracted from deparsing a plan node |
 
 ## Functions
@@ -95,14 +101,15 @@ The statistics of the pg_tracing extension itself are tracked and made available
 
 | Column | Type | Description |
 | --- | --- | --- 
-| `traces` | bigint | Total number of traces captured |
-| `spans` | bigint | Total number of spans captured |
+| `processed_traces` | bigint | Total number of traces processed |
+| `processed_spans` | bigint | Total number of spans processed |
+| `dropped_traces` | bigint | Total number of traces dropped due to exceeding `pg_tracing.max_span` spans |
 | `dropped_spans` | bigint | Total number of spans dropped due to exceeding `pg_tracing.max_span` spans |
-| `failed_truncates` | bigint | Total number of times the module couldn't truncate the query file due to conflict lock on pg_tracing's text file |
+| `otel_sent_spans` | bigint | Total number of spans dropped successfully sent to the otel collector |
+| `otel_failures` | bigint | Total number of failures to send spans to the otel collector  |
 | `last_consume` | timestamp with time zone | Time at which spans were last consumed |
 | `stats_reset` | timestamp with time zone | Time at which all statistics in the `pg_tracing_info` view were last reset |
 
-
 ### pg_tracing_reset()
 
 Discards all statistics gathered so far by `pg_tracing`. Span buffer is not emptied by this function. By default, this function can only be executed by superusers. Access may be granted to others using `GRANT`.

diff --git a/expected/guc.out b/expected/guc.out
@@ -1,13 +1,23 @@
 -- Test trace context propagation through GUCs
-SET pg_tracing.trace_context='dddbs=''postgres.db'',traceparent=''00-00000000000000000000000000000004-0000000000000004-01';
+SET pg_tracing.trace_context='dddbs=''postgres.db'',traceparent=''00-00000000000000000000000000000004-0000000000000004-01''';
 SELECT 1;
  ?column? 
 ----------
         1
 (1 row)
 
+-- Test trace context propagation through a local GUCs
+BEGIN;
+SET LOCAL pg_tracing.trace_context='dddbs=''postgres.db'',traceparent=''00-00000000000000000000000000000005-0000000000000005-01''';
+SELECT 1;
+ ?column? 
+----------
+        1
+(1 row)
+
+COMMIT;
 -- Test multiple statements
-SET pg_tracing.trace_context='dddbs=''postgres.db'',traceparent=''00-fffffffffffffffffffffffffffffff5-0000000000000005-01';
+SET pg_tracing.trace_context='dddbs=''postgres.db'',traceparent=''00-fffffffffffffffffffffffffffffff5-0000000000000005-01''';
 SELECT 2;
  ?column? 
 ----------
@@ -28,6 +38,12 @@ select trace_id, span_operation, parameters, lvl from peek_ordered_spans;
  00000000000000000000000000000004 | Planner        |            |   2
  00000000000000000000000000000004 | ExecutorRun    |            |   2
  00000000000000000000000000000004 | Result         |            |   3
+ 00000000000000000000000000000005 | SELECT $1;     | {1}        |   1
+ 00000000000000000000000000000005 | Planner        |            |   2
+ 00000000000000000000000000000005 | ExecutorRun    |            |   2
+ 00000000000000000000000000000005 | Result         |            |   3
+ 00000000000000000000000000000005 | COMMIT;        |            |   1
+ 00000000000000000000000000000005 | ProcessUtility |            |   2
  fffffffffffffffffffffffffffffff5 | SELECT $1;     | {2}        |   1
  fffffffffffffffffffffffffffffff5 | Planner        |            |   2
  fffffffffffffffffffffffffffffff5 | ExecutorRun    |            |   2
@@ -36,11 +52,11 @@ select trace_id, span_operation, parameters, lvl from peek_ordered_spans;
  fffffffffffffffffffffffffffffff5 | Planner        |            |   2
  fffffffffffffffffffffffffffffff5 | ExecutorRun    |            |   2
  fffffffffffffffffffffffffffffff5 | Result         |            |   3
-(12 rows)
+(18 rows)
 
 CALL clean_spans();
 -- Mix SQLCommenter and GUC propagation
-SET pg_tracing.trace_context='dddbs=''postgres.db'',traceparent=''00-fffffffffffffffffffffffffffffff6-0000000000000006-01';
+SET pg_tracing.trace_context='dddbs=''postgres.db'',traceparent=''00-fffffffffffffffffffffffffffffff6-0000000000000006-01''';
 SELECT 2;
  ?column? 
 ----------
@@ -97,17 +113,21 @@ SELECT 4;
 (1 row)
 
 -- Test no traceparent field
-SET pg_tracing.trace_context='dddbs=''postgres.db'',taceparent=''00-fffffffffffffffffffffffffffffff5-0000000000000005-01';
-ERROR:  invalid value for parameter "pg_tracing.trace_context": "dddbs='postgres.db',taceparent='00-fffffffffffffffffffffffffffffff5-0000000000000005-01"
+SET pg_tracing.trace_context='dddbs=''postgres.db'',taceparent=''00-fffffffffffffffffffffffffffffff5-0000000000000005-01''';
+ERROR:  invalid value for parameter "pg_tracing.trace_context": "dddbs='postgres.db',taceparent='00-fffffffffffffffffffffffffffffff5-0000000000000005-01'"
 DETAIL:  Error parsing tracecontext: No traceparent field found
 -- Test incorrect trace id
-SET pg_tracing.trace_context='dddbs=''postgres.db'',traceparent=''00-ffffffffffffffffffffffffffffff5-0000000000000005-01';
-ERROR:  invalid value for parameter "pg_tracing.trace_context": "dddbs='postgres.db',traceparent='00-ffffffffffffffffffffffffffffff5-0000000000000005-01"
+SET pg_tracing.trace_context='dddbs=''postgres.db'',traceparent=''00-ffffffffffffffffffffffffffffff5-0000000000000005-01''';
+ERROR:  invalid value for parameter "pg_tracing.trace_context": "dddbs='postgres.db',traceparent='00-ffffffffffffffffffffffffffffff5-0000000000000005-01'"
 DETAIL:  Error parsing tracecontext: Traceparent field doesn't have the correct size
 -- Test wrong format
-SET pg_tracing.trace_context='dddbs=''postgres.db'',traceparent=''00f-ffffffffffffffffffffffffffffff5-0000000000000005-01';
-ERROR:  invalid value for parameter "pg_tracing.trace_context": "dddbs='postgres.db',traceparent='00f-ffffffffffffffffffffffffffffff5-0000000000000005-01"
+SET pg_tracing.trace_context='dddbs=''postgres.db'',traceparent=''00f-ffffffffffffffffffffffffffffff5-0000000000000005-01''';
+ERROR:  invalid value for parameter "pg_tracing.trace_context": "dddbs='postgres.db',traceparent='00f-ffffffffffffffffffffffffffffff5-0000000000000005-01'"
 DETAIL:  Error parsing tracecontext: Incorrect traceparent format
+-- Test missing end quote
+SET pg_tracing.trace_context='dddbs=''postgres.db'',traceparent=''00-fffffffffffffffffffffffffffffff6-0000000000000006-01';
+ERROR:  invalid value for parameter "pg_tracing.trace_context": "dddbs='postgres.db',traceparent='00-fffffffffffffffffffffffffffffff6-0000000000000006-01"
+DETAIL:  Error parsing tracecontext: Traceparent field doesn't have the correct size
 -- GUC errors and no GUC tracecontext should not generate spans
 select count(*) = 0 from peek_ordered_spans;
  ?column? 

diff --git a/expected/insert.out b/expected/insert.out
@@ -3,26 +3,26 @@ SET pg_tracing.sample_rate = 0.0;
 SET pg_tracing.caller_sample_rate = 1.0;
 /*dddbs='postgres.db',traceparent='00-00000000000000000000000000000001-0000000000000001-01'*/ CREATE TABLE IF NOT EXISTS pg_tracing_test_table_with_constraint (a int, b char(20), CONSTRAINT PK_tracing_test PRIMARY KEY (a));
 SELECT span_type, span_operation, lvl from peek_ordered_spans where trace_id='00000000000000000000000000000001';
-   span_type    |                                                          span_operation                                                           | lvl 
-----------------+-----------------------------------------------------------------------------------------------------------------------------------+-----
- Utility query  | CREATE TABLE IF NOT EXISTS pg_tracing_test_table_with_constraint (a int, b char(20), CONSTRAINT PK_tracing_test PRIMARY KEY (a)); |   1
- ProcessUtility | ProcessUtility                                                                                                                    |   2
- Utility query  | CREATE TABLE IF NOT EXISTS pg_tracing_test_table_with_constraint (a int, b char(20), CONSTRAINT PK_tracing_test PRIMARY KEY (a)); |   3
- ProcessUtility | ProcessUtility                                                                                                                    |   4
- Commit         | Commit                                                                                                                            |   1
+     span_type     |                                                          span_operation                                                           | lvl 
+-------------------+-----------------------------------------------------------------------------------------------------------------------------------+-----
+ Utility query     | CREATE TABLE IF NOT EXISTS pg_tracing_test_table_with_constraint (a int, b char(20), CONSTRAINT PK_tracing_test PRIMARY KEY (a)); |   1
+ ProcessUtility    | ProcessUtility                                                                                                                    |   2
+ Utility query     | CREATE TABLE IF NOT EXISTS pg_tracing_test_table_with_constraint (a int, b char(20), CONSTRAINT PK_tracing_test PRIMARY KEY (a)); |   3
+ ProcessUtility    | ProcessUtility                                                                                                                    |   4
+ TransactionCommit | TransactionCommit                                                                                                                 |   1
 (5 rows)
 
 -- Simple insertion
 /*dddbs='postgres.db',traceparent='00-00000000000000000000000000000002-0000000000000002-01'*/ INSERT INTO pg_tracing_test_table_with_constraint VALUES(1, 'aaa');
 SELECT span_type, span_operation from peek_ordered_spans where trace_id='00000000000000000000000000000002';
-  span_type   |                          span_operation                           
---------------+-------------------------------------------------------------------
- Insert query | INSERT INTO pg_tracing_test_table_with_constraint VALUES($1, $2);
- Planner      | Planner
- ExecutorRun  | ExecutorRun
- Insert       | Insert on pg_tracing_test_table_with_constraint
- Result       | Result
- Commit       | Commit
+     span_type     |                          span_operation                           
+-------------------+-------------------------------------------------------------------
+ Insert query      | INSERT INTO pg_tracing_test_table_with_constraint VALUES($1, $2);
+ Planner           | Planner
+ ExecutorRun       | ExecutorRun
+ Insert            | Insert on pg_tracing_test_table_with_constraint
+ Result            | Result
+ TransactionCommit | TransactionCommit
 (6 rows)
 
 -- Trigger constraint violation