pg_fact_loader
pg_fact_loader
pg_fact_loader : build fact tables with Postgres
Overview
| ID | Extension | Package | Version | Category | License | Language |
|---|---|---|---|---|---|---|
| 9820 | pg_fact_loader
|
pg_fact_loader
|
2.0.1 |
ETL
|
MIT
|
C
|
| Attribute | Has Binary | Has Library | Need Load | Has DDL | Relocatable | Trusted |
|---|---|---|---|---|---|---|
--s-d--
|
No
|
Yes
|
No
|
Yes
|
no
|
no
|
| Relationships | |
|---|---|
| Schemas | fact_loader |
| See Also | pg_cron
pg_partman
pg_jobmon
mimeo
timescaledb
citus
tablefunc
pg_bulkload
|
Packages
| Type | Repo | Version | PG Major Compatibility | Package Pattern | Dependencies |
|---|---|---|---|---|---|
| EXT | PGDG
|
2.0.1 |
18
17
16
15
14
|
pg_fact_loader |
- |
| RPM | PGDG
|
2.0.1 |
18
17
16
15
14
|
pg_fact_loader_$v |
- |
| DEB | PGDG
|
2.0.1 |
18
17
16
15
14
|
postgresql-$v-pg-fact-loader |
- |
| Linux / PG | PG18 | PG17 | PG16 | PG15 | PG14 |
|---|---|---|---|---|---|
el8.x86_64
|
PGDG 2.0.1
|
PGDG 2.0.1
|
PGDG 2.0.1
|
PGDG 2.0.1
|
PGDG 2.0.1
|
el8.aarch64
|
PGDG 2.0.1
|
PGDG 2.0.1
|
PGDG 2.0.1
|
PGDG 2.0.1
|
PGDG 2.0.1
|
el9.x86_64
|
PGDG 2.0.1
|
PGDG 2.0.1
|
PGDG 2.0.1
|
PGDG 2.0.1
|
PGDG 2.0.1
|
el9.aarch64
|
PGDG 2.0.1
|
PGDG 2.0.1
|
PGDG 2.0.1
|
PGDG 2.0.1
|
PGDG 2.0.1
|
el10.x86_64
|
PGDG 2.0.1
|
PGDG 2.0.1
|
PGDG 2.0.1
|
PGDG 2.0.1
|
PGDG 2.0.1
|
el10.aarch64
|
PGDG 2.0.1
|
PGDG 2.0.1
|
PGDG 2.0.1
|
PGDG 2.0.1
|
PGDG 2.0.1
|
d12.x86_64
|
MISS
|
PGDG 2.0.1
|
PGDG 2.0.1
|
PGDG 2.0.1
|
PGDG 2.0.1
|
d12.aarch64
|
MISS
|
PGDG 2.0.1
|
PGDG 2.0.1
|
PGDG 2.0.1
|
PGDG 2.0.1
|
d13.x86_64
|
MISS
|
PGDG 2.0.1
|
PGDG 2.0.1
|
PGDG 2.0.1
|
PGDG 2.0.1
|
d13.aarch64
|
MISS
|
PGDG 2.0.1
|
PGDG 2.0.1
|
PGDG 2.0.1
|
PGDG 2.0.1
|
u22.x86_64
|
MISS
|
PGDG 2.0.1
|
PGDG 2.0.1
|
PGDG 2.0.1
|
PGDG 2.0.1
|
u22.aarch64
|
MISS
|
PGDG 2.0.1
|
PGDG 2.0.1
|
PGDG 2.0.1
|
PGDG 2.0.1
|
u24.x86_64
|
MISS
|
PGDG 2.0.1
|
PGDG 2.0.1
|
PGDG 2.0.1
|
PGDG 2.0.1
|
u24.aarch64
|
MISS
|
PGDG 2.0.1
|
PGDG 2.0.1
|
PGDG 2.0.1
|
PGDG 2.0.1
|
| Package | Version | OS | ORG | SIZE | File URL |
|---|---|---|---|---|---|
pg_fact_loader_18 |
2.0.1 |
el8.x86_64 | pgdg | 36.2 KiB | pg_fact_loader_18-2.0.1-3PGDG.rhel8.noarch.rpm |
pg_fact_loader_18 |
2.0.1 |
el8.aarch64 | pgdg | 36.1 KiB | pg_fact_loader_18-2.0.1-3PGDG.rhel8.noarch.rpm |
pg_fact_loader_18 |
2.0.1 |
el9.x86_64 | pgdg | 34.6 KiB | pg_fact_loader_18-2.0.1-3PGDG.rhel9.noarch.rpm |
pg_fact_loader_18 |
2.0.1 |
el9.aarch64 | pgdg | 34.5 KiB | pg_fact_loader_18-2.0.1-3PGDG.rhel9.noarch.rpm |
pg_fact_loader_18 |
2.0.1 |
el10.x86_64 | pgdg | 35.1 KiB | pg_fact_loader_18-2.0.1-3PGDG.rhel10.noarch.rpm |
pg_fact_loader_18 |
2.0.1 |
el10.aarch64 | pgdg | 35.0 KiB | pg_fact_loader_18-2.0.1-3PGDG.rhel10.noarch.rpm |
Source
Install
Make sure PGDG repo available:
pig repo add pgdg -u # add pgdg repo and update cacheInstall this extension with pig:
pig install pg_fact_loader; # install via package name, for the active PG version
pig install pg_fact_loader -v 18; # install for PG 18
pig install pg_fact_loader -v 17; # install for PG 17
pig install pg_fact_loader -v 16; # install for PG 16
pig install pg_fact_loader -v 15; # install for PG 15
pig install pg_fact_loader -v 14; # install for PG 14Create this extension with:
CREATE EXTENSION pg_fact_loader;Usage
Build and maintain fact tables using queue-based change data capture. Processes audit/change log tables to incrementally update fact tables.
Enabling
CREATE EXTENSION pg_fact_loader;Optionally with pglogical for replica-based setup:
CREATE EXTENSION pglogical;
CREATE EXTENSION pglogical_ticker;
CREATE EXTENSION pg_fact_loader;Workflow
- Replicate source tables to a reporting database (via pglogical or other means)
- Create audit/change log tables on the OLTP system for source tables
- Create a fact table structure for aggregated data
- Create a merge function that takes a key ID and returns one row of the fact table
- Configure pg_fact_loader to wire queue tables to fact tables
- Backfill the fact table initially
- Schedule the worker to process changes continuously
Configuration Tables
-- Register a fact table
INSERT INTO fact_loader.fact_tables (fact_table_relid, fact_table_agg_proid, ...)
VALUES ('public.customers_fact'::regclass, 'customers_fact_merge'::regproc, ...);
-- Register queue (audit) tables
INSERT INTO fact_loader.queue_tables (queue_table_relid, queue_of_base_table_relid, ...)
VALUES ('audit.customers_audit'::regclass, 'public.customers'::regclass, ...);
-- Connect queue tables to fact tables with merge functions
INSERT INTO fact_loader.queue_table_deps
(fact_table_id, queue_table_id, insert_merge_proid, update_merge_proid, delete_merge_proid)
VALUES (1, 1, 'customers_fact_merge'::regproc, 'customers_fact_merge'::regproc, 'customers_fact_merge'::regproc);
-- Define how to retrieve the key from queue entries
INSERT INTO fact_loader.key_retrieval_sequences
(queue_table_dep_id, return_columns, is_fact_key)
VALUES (1, '{customer_id}', true);Running the Worker
-- Process pending changes
SELECT fact_loader.worker();
-- Schedule this to run periodically (e.g., every few seconds via pg_cron)Initial Backfill
-- Run the merge function for every existing row
SELECT customers_fact_merge(customer_id) FROM customers;Adding Batch ID Fields
SELECT fact_loader.add_batch_id_fields();Key Features
- Queue-based incremental fact table updates
- Supports insert, update, and delete events
- Handles multi-level key retrieval (joins through multiple tables)
- Fact table dependency chains (child facts updated after parent)
- Checks replication lag before processing (when used with pglogical)
Last updated on