pgspider_ext

pgspider_ext

pgspider_ext : foreign-data wrapper for remote PGSpider servers

Overview

ID Extension Package Version Category License Language
8540
pgspider_ext
pgspider_ext
1.3.0
FDW
PostgreSQL
C
Attribute Has Binary Has Library Need Load Has DDL Relocatable Trusted
--s-d-r
No
Yes
No
Yes
yes
no
Relationships
See Also
plproxy
wrappers
multicorn
postgres_fdw
citus
mysql_fdw
oracle_fdw
mongo_fdw

Packages

Type Repo Version PG Major Compatibility Package Pattern Dependencies
EXT
PGDG
1.3.0
18
17
16
15
14
pgspider_ext -
RPM
PGDG
1.3.0
18
17
16
15
14
pgspider_ext_$v -
DEB
PIGSTY
1.3.0
18
17
16
15
14
postgresql-$v-pgspider-ext -
Linux / PG PG18 PG17 PG16 PG15 PG14
el8.x86_64
MISS
PGDG 1.3.0
PGDG 1.3.0
PGDG 1.3.0
MISS
el8.aarch64
MISS
PGDG 1.3.0
PGDG 1.3.0
PGDG 1.3.0
MISS
el9.x86_64
MISS
PGDG 1.3.0
PGDG 1.3.0
PGDG 1.3.0
MISS
el9.aarch64
MISS
PGDG 1.3.0
PGDG 1.3.0
PGDG 1.3.0
MISS
el10.x86_64
MISS
PGDG 1.3.0
PGDG 1.3.0
PGDG 1.3.0
MISS
el10.aarch64
MISS
PGDG 1.3.0
PGDG 1.3.0
PGDG 1.3.0
MISS
d12.x86_64
MISS
PIGSTY 1.3.0
PIGSTY 1.3.0
PIGSTY 1.3.0
MISS
d12.aarch64
MISS
PIGSTY 1.3.0
PIGSTY 1.3.0
PIGSTY 1.3.0
MISS
d13.x86_64
MISS
PIGSTY 1.3.0
PIGSTY 1.3.0
PIGSTY 1.3.0
MISS
d13.aarch64
MISS
PIGSTY 1.3.0
PIGSTY 1.3.0
PIGSTY 1.3.0
MISS
u22.x86_64
MISS
PIGSTY 1.3.0
PIGSTY 1.3.0
PIGSTY 1.3.0
MISS
u22.aarch64
MISS
PIGSTY 1.3.0
PIGSTY 1.3.0
PIGSTY 1.3.0
MISS
u24.x86_64
MISS
PIGSTY 1.3.0
PIGSTY 1.3.0
PIGSTY 1.3.0
MISS
u24.aarch64
MISS
PIGSTY 1.3.0
PIGSTY 1.3.0
PIGSTY 1.3.0
MISS
Package Version OS ORG SIZE File URL
pgspider_ext_17 1.3.0 el8.x86_64 pgdg 28.6 KiB pgspider_ext_17-1.3.0-1PGDG.rhel8.x86_64.rpm
pgspider_ext_17 1.3.0 el8.aarch64 pgdg 27.7 KiB pgspider_ext_17-1.3.0-1PGDG.rhel8.aarch64.rpm
pgspider_ext_17 1.3.0 el9.x86_64 pgdg 29.1 KiB pgspider_ext_17-1.3.0-1PGDG.rhel9.x86_64.rpm
pgspider_ext_17 1.3.0 el9.aarch64 pgdg 28.4 KiB pgspider_ext_17-1.3.0-1PGDG.rhel9.aarch64.rpm
pgspider_ext_17 1.3.0 el10.x86_64 pgdg 30.1 KiB pgspider_ext_17-1.3.0-1PGDG.rhel10.x86_64.rpm
pgspider_ext_17 1.3.0 el10.aarch64 pgdg 29.1 KiB pgspider_ext_17-1.3.0-1PGDG.rhel10.aarch64.rpm
postgresql-17-pgspider-ext 1.3.0 d12.x86_64 pigsty 48.8 KiB postgresql-17-pgspider-ext_1.3.0-1PIGSTY~bookworm_amd64.deb
postgresql-17-pgspider-ext 1.3.0 d12.aarch64 pigsty 47.1 KiB postgresql-17-pgspider-ext_1.3.0-1PIGSTY~bookworm_arm64.deb
postgresql-17-pgspider-ext 1.3.0 d13.x86_64 pigsty 48.8 KiB postgresql-17-pgspider-ext_1.3.0-1PIGSTY~trixie_amd64.deb
postgresql-17-pgspider-ext 1.3.0 d13.aarch64 pigsty 47.3 KiB postgresql-17-pgspider-ext_1.3.0-1PIGSTY~trixie_arm64.deb
postgresql-17-pgspider-ext 1.3.0 u22.x86_64 pigsty 61.7 KiB postgresql-17-pgspider-ext_1.3.0-1PIGSTY~jammy_amd64.deb
postgresql-17-pgspider-ext 1.3.0 u22.aarch64 pigsty 60.7 KiB postgresql-17-pgspider-ext_1.3.0-1PIGSTY~jammy_arm64.deb
postgresql-17-pgspider-ext 1.3.0 u24.x86_64 pigsty 50.9 KiB postgresql-17-pgspider-ext_1.3.0-1PIGSTY~noble_amd64.deb
postgresql-17-pgspider-ext 1.3.0 u24.aarch64 pigsty 49.6 KiB postgresql-17-pgspider-ext_1.3.0-1PIGSTY~noble_arm64.deb
Package Version OS ORG SIZE File URL
pgspider_ext_16 1.3.0 el8.x86_64 pgdg 28.6 KiB pgspider_ext_16-1.3.0-1PGDG.rhel8.x86_64.rpm
pgspider_ext_16 1.3.0 el8.aarch64 pgdg 27.8 KiB pgspider_ext_16-1.3.0-1PGDG.rhel8.aarch64.rpm
pgspider_ext_16 1.3.0 el9.x86_64 pgdg 29.2 KiB pgspider_ext_16-1.3.0-1PGDG.rhel9.x86_64.rpm
pgspider_ext_16 1.3.0 el9.aarch64 pgdg 28.5 KiB pgspider_ext_16-1.3.0-1PGDG.rhel9.aarch64.rpm
pgspider_ext_16 1.3.0 el10.x86_64 pgdg 30.2 KiB pgspider_ext_16-1.3.0-1PGDG.rhel10.x86_64.rpm
pgspider_ext_16 1.3.0 el10.aarch64 pgdg 29.2 KiB pgspider_ext_16-1.3.0-1PGDG.rhel10.aarch64.rpm
postgresql-16-pgspider-ext 1.3.0 d12.x86_64 pigsty 48.7 KiB postgresql-16-pgspider-ext_1.3.0-1PIGSTY~bookworm_amd64.deb
postgresql-16-pgspider-ext 1.3.0 d12.aarch64 pigsty 47.0 KiB postgresql-16-pgspider-ext_1.3.0-1PIGSTY~bookworm_arm64.deb
postgresql-16-pgspider-ext 1.3.0 d13.x86_64 pigsty 48.8 KiB postgresql-16-pgspider-ext_1.3.0-1PIGSTY~trixie_amd64.deb
postgresql-16-pgspider-ext 1.3.0 d13.aarch64 pigsty 47.2 KiB postgresql-16-pgspider-ext_1.3.0-1PIGSTY~trixie_arm64.deb
postgresql-16-pgspider-ext 1.3.0 u22.x86_64 pigsty 61.3 KiB postgresql-16-pgspider-ext_1.3.0-1PIGSTY~jammy_amd64.deb
postgresql-16-pgspider-ext 1.3.0 u22.aarch64 pigsty 60.3 KiB postgresql-16-pgspider-ext_1.3.0-1PIGSTY~jammy_arm64.deb
postgresql-16-pgspider-ext 1.3.0 u24.x86_64 pigsty 50.8 KiB postgresql-16-pgspider-ext_1.3.0-1PIGSTY~noble_amd64.deb
postgresql-16-pgspider-ext 1.3.0 u24.aarch64 pigsty 49.6 KiB postgresql-16-pgspider-ext_1.3.0-1PIGSTY~noble_arm64.deb
Package Version OS ORG SIZE File URL
pgspider_ext_15 1.3.0 el8.x86_64 pgdg 29.0 KiB pgspider_ext_15-1.3.0-1PGDG.rhel8.x86_64.rpm
pgspider_ext_15 1.3.0 el8.aarch64 pgdg 28.0 KiB pgspider_ext_15-1.3.0-1PGDG.rhel8.aarch64.rpm
pgspider_ext_15 1.3.0 el9.x86_64 pgdg 29.6 KiB pgspider_ext_15-1.3.0-1PGDG.rhel9.x86_64.rpm
pgspider_ext_15 1.3.0 el9.aarch64 pgdg 28.8 KiB pgspider_ext_15-1.3.0-1PGDG.rhel9.aarch64.rpm
pgspider_ext_15 1.3.0 el10.x86_64 pgdg 30.4 KiB pgspider_ext_15-1.3.0-1PGDG.rhel10.x86_64.rpm
pgspider_ext_15 1.3.0 el10.aarch64 pgdg 29.6 KiB pgspider_ext_15-1.3.0-1PGDG.rhel10.aarch64.rpm
postgresql-15-pgspider-ext 1.3.0 d12.x86_64 pigsty 48.9 KiB postgresql-15-pgspider-ext_1.3.0-1PIGSTY~bookworm_amd64.deb
postgresql-15-pgspider-ext 1.3.0 d12.aarch64 pigsty 47.0 KiB postgresql-15-pgspider-ext_1.3.0-1PIGSTY~bookworm_arm64.deb
postgresql-15-pgspider-ext 1.3.0 d13.x86_64 pigsty 49.0 KiB postgresql-15-pgspider-ext_1.3.0-1PIGSTY~trixie_amd64.deb
postgresql-15-pgspider-ext 1.3.0 d13.aarch64 pigsty 47.3 KiB postgresql-15-pgspider-ext_1.3.0-1PIGSTY~trixie_arm64.deb
postgresql-15-pgspider-ext 1.3.0 u22.x86_64 pigsty 61.3 KiB postgresql-15-pgspider-ext_1.3.0-1PIGSTY~jammy_amd64.deb
postgresql-15-pgspider-ext 1.3.0 u22.aarch64 pigsty 60.3 KiB postgresql-15-pgspider-ext_1.3.0-1PIGSTY~jammy_arm64.deb
postgresql-15-pgspider-ext 1.3.0 u24.x86_64 pigsty 51.0 KiB postgresql-15-pgspider-ext_1.3.0-1PIGSTY~noble_amd64.deb
postgresql-15-pgspider-ext 1.3.0 u24.aarch64 pigsty 49.7 KiB postgresql-15-pgspider-ext_1.3.0-1PIGSTY~noble_arm64.deb

Source

pig build pkg pgspider_ext;		# build deb

Install

Make sure PGDG repo available:

pig repo add pgdg -u    # add pgdg repo and update cache

Install this extension with pig:

pig install pgspider_ext;		# install via package name, for the active PG version

pig install pgspider_ext -v 17;   # install for PG 17
pig install pgspider_ext -v 16;   # install for PG 16
pig install pgspider_ext -v 15;   # install for PG 15

Create this extension with:

CREATE EXTENSION pgspider_ext;

Usage

pgspider_ext: Foreign data wrapper for remote PGSpider servers

PGSpider Extension turns PostgreSQL into a distributed query engine by creating virtual partitioned tables that transparently query data across multiple remote nodes in parallel.

Setup Child Servers

First, create servers for each data source using their respective FDWs:

CREATE EXTENSION pgspider_ext;
CREATE EXTENSION postgres_fdw;

CREATE SERVER pgsrv1 FOREIGN DATA WRAPPER postgres_fdw
  OPTIONS (host '127.0.0.1', port '5433', dbname 'postgres');
CREATE SERVER pgsrv2 FOREIGN DATA WRAPPER postgres_fdw
  OPTIONS (host '127.0.0.1', port '5434', dbname 'postgres');

CREATE USER MAPPING FOR CURRENT_USER SERVER pgsrv1
  OPTIONS (user 'user', password 'pass');
CREATE USER MAPPING FOR CURRENT_USER SERVER pgsrv2
  OPTIONS (user 'user', password 'pass');

Create Child Foreign Tables

CREATE FOREIGN TABLE t1_pg1_child (i int, t text)
  SERVER pgsrv1 OPTIONS (table_name 't1');
CREATE FOREIGN TABLE t1_pg2_child (i int, t text)
  SERVER pgsrv2 OPTIONS (table_name 't1');

Create PGSpider Partitioned Table

Create a PGSpider server and a partitioned parent table with an extra partition key column:

CREATE SERVER spdsrv FOREIGN DATA WRAPPER pgspider_ext;
CREATE USER MAPPING FOR CURRENT_USER SERVER spdsrv;

CREATE TABLE t1 (i int, t text, node text)
  PARTITION BY LIST (node);

CREATE FOREIGN TABLE t1_pg1 PARTITION OF t1
  FOR VALUES IN ('node1') SERVER spdsrv;
CREATE FOREIGN TABLE t1_pg2 PARTITION OF t1
  FOR VALUES IN ('node2') SERVER spdsrv
  OPTIONS (child_name 't1_pg2_child');

By default, PGSpider finds child tables using the pattern [parent_table_name]_child. Use child_name to specify a different name.

Query Across All Nodes

SELECT * FROM t1;
 i  | t | node
----+---+-------
 10 | a | node1
 11 | b | node1
 20 | c | node2
 21 | d | node2

Queries automatically fan out to all child nodes in parallel. WHERE clauses and aggregate functions are pushed down to child nodes:

SET enable_partitionwise_aggregate TO on;
SELECT count(*), node FROM t1 GROUP BY node;

Note: Only SELECT operations are supported; INSERT, UPDATE, and DELETE are not supported.

Last updated on