Extensions
pg_tiktoken

pg_tiktoken

pg_tiktoken : tiktoken tokenizer for use with OpenAI models in postgres

Overview

ID Extension Package Version Category License Language
1870
pg_tiktoken
pg_tiktoken
 0.0.1
RAG
Apache-2.0
Rust
Attribute Has Binary Has Library Need Load Has DDL Relocatable Trusted
--s-d--
No
Yes
No
Yes
no
no
Relationships
See Also
vectorize
pg_summarize
pg4ml
pgml
vector
vchord
vectorscale
pg_graphql

Packages

Type Repo Version PG Major Compatibility Package Pattern Dependencies
EXT
PIGSTY
 0.0.1
18
17
16
15
14
13
 pg_tiktoken -
RPM
PIGSTY
 0.0.1
18
17
16
15
14
13
 pg_tiktoken_$v -
DEB
PIGSTY
 0.0.1
18
17
16
15
14
13
 postgresql-$v-pg-tiktoken -
Linux / PG PG18 PG17 PG16 PG15 PG14 PG13
el8.x86_64
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
el8.aarch64
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
el9.x86_64
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
el9.aarch64
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
el10.x86_64
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
el10.aarch64
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
d12.x86_64
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
d12.aarch64
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
d13.x86_64
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
d13.aarch64
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
u22.x86_64
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
u22.aarch64
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
u24.x86_64
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
u24.aarch64
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
Package Version OS ORG SIZE File URL
pg_tiktoken_18 0.0.1 el8.x86_64 pigsty 1.7 MiB pg_tiktoken_18-0.0.1-2PIGSTY.el8.x86_64.rpm
pg_tiktoken_18 0.0.1 el8.aarch64 pigsty 1.6 MiB pg_tiktoken_18-0.0.1-2PIGSTY.el8.aarch64.rpm
pg_tiktoken_18 0.0.1 el9.x86_64 pigsty 1.6 MiB pg_tiktoken_18-0.0.1-2PIGSTY.el9.x86_64.rpm
pg_tiktoken_18 0.0.1 el9.aarch64 pigsty 1.5 MiB pg_tiktoken_18-0.0.1-2PIGSTY.el9.aarch64.rpm
pg_tiktoken_18 0.0.1 el10.x86_64 pigsty 1.7 MiB pg_tiktoken_18-0.0.1-2PIGSTY.el10.x86_64.rpm
pg_tiktoken_18 0.0.1 el10.aarch64 pigsty 1.5 MiB pg_tiktoken_18-0.0.1-2PIGSTY.el10.aarch64.rpm
postgresql-18-pg-tiktoken 0.0.1 d12.x86_64 pigsty 5.0 KiB postgresql-18-pg-tiktoken_0.0.1-1PIGSTY~bookworm_amd64.deb
postgresql-18-pg-tiktoken 0.0.1 d12.aarch64 pigsty 5.0 KiB postgresql-18-pg-tiktoken_0.0.1-1PIGSTY~bookworm_arm64.deb
postgresql-18-pg-tiktoken 0.0.1 d13.x86_64 pigsty 4.9 KiB postgresql-18-pg-tiktoken_0.0.1-1PIGSTY~trixie_amd64.deb
postgresql-18-pg-tiktoken 0.0.1 d13.aarch64 pigsty 4.9 KiB postgresql-18-pg-tiktoken_0.0.1-1PIGSTY~trixie_arm64.deb
postgresql-18-pg-tiktoken 0.0.1 u22.x86_64 pigsty 4.9 KiB postgresql-18-pg-tiktoken_0.0.1-1PIGSTY~jammy_amd64.deb
postgresql-18-pg-tiktoken 0.0.1 u22.aarch64 pigsty 4.9 KiB postgresql-18-pg-tiktoken_0.0.1-1PIGSTY~jammy_arm64.deb
postgresql-18-pg-tiktoken 0.0.1 u24.x86_64 pigsty 4.9 KiB postgresql-18-pg-tiktoken_0.0.1-1PIGSTY~noble_amd64.deb
postgresql-18-pg-tiktoken 0.0.1 u24.aarch64 pigsty 4.9 KiB postgresql-18-pg-tiktoken_0.0.1-1PIGSTY~noble_arm64.deb
Package Version OS ORG SIZE File URL
pg_tiktoken_17 0.0.1 el8.x86_64 pigsty 1.7 MiB pg_tiktoken_17-0.0.1-2PIGSTY.el8.x86_64.rpm
pg_tiktoken_17 0.0.1 el8.aarch64 pigsty 1.6 MiB pg_tiktoken_17-0.0.1-2PIGSTY.el8.aarch64.rpm
pg_tiktoken_17 0.0.1 el9.x86_64 pigsty 1.6 MiB pg_tiktoken_17-0.0.1-2PIGSTY.el9.x86_64.rpm
pg_tiktoken_17 0.0.1 el9.aarch64 pigsty 1.5 MiB pg_tiktoken_17-0.0.1-2PIGSTY.el9.aarch64.rpm
pg_tiktoken_17 0.0.1 el10.x86_64 pigsty 1.7 MiB pg_tiktoken_17-0.0.1-2PIGSTY.el10.x86_64.rpm
pg_tiktoken_17 0.0.1 el10.aarch64 pigsty 1.5 MiB pg_tiktoken_17-0.0.1-2PIGSTY.el10.aarch64.rpm
postgresql-17-pg-tiktoken 0.0.1 d12.x86_64 pigsty 5.0 KiB postgresql-17-pg-tiktoken_0.0.1-1PIGSTY~bookworm_amd64.deb
postgresql-17-pg-tiktoken 0.0.1 d12.aarch64 pigsty 5.0 KiB postgresql-17-pg-tiktoken_0.0.1-1PIGSTY~bookworm_arm64.deb
postgresql-17-pg-tiktoken 0.0.1 d13.x86_64 pigsty 4.9 KiB postgresql-17-pg-tiktoken_0.0.1-1PIGSTY~trixie_amd64.deb
postgresql-17-pg-tiktoken 0.0.1 d13.aarch64 pigsty 4.9 KiB postgresql-17-pg-tiktoken_0.0.1-1PIGSTY~trixie_arm64.deb
postgresql-17-pg-tiktoken 0.0.1 u22.x86_64 pigsty 4.9 KiB postgresql-17-pg-tiktoken_0.0.1-1PIGSTY~jammy_amd64.deb
postgresql-17-pg-tiktoken 0.0.1 u22.aarch64 pigsty 4.9 KiB postgresql-17-pg-tiktoken_0.0.1-1PIGSTY~jammy_arm64.deb
postgresql-17-pg-tiktoken 0.0.1 u24.x86_64 pigsty 4.9 KiB postgresql-17-pg-tiktoken_0.0.1-1PIGSTY~noble_amd64.deb
postgresql-17-pg-tiktoken 0.0.1 u24.aarch64 pigsty 4.9 KiB postgresql-17-pg-tiktoken_0.0.1-1PIGSTY~noble_arm64.deb
Package Version OS ORG SIZE File URL
pg_tiktoken_16 0.0.1 el8.x86_64 pigsty 1.7 MiB pg_tiktoken_16-0.0.1-2PIGSTY.el8.x86_64.rpm
pg_tiktoken_16 0.0.1 el8.aarch64 pigsty 1.6 MiB pg_tiktoken_16-0.0.1-2PIGSTY.el8.aarch64.rpm
pg_tiktoken_16 0.0.1 el9.x86_64 pigsty 1.6 MiB pg_tiktoken_16-0.0.1-2PIGSTY.el9.x86_64.rpm
pg_tiktoken_16 0.0.1 el9.aarch64 pigsty 1.5 MiB pg_tiktoken_16-0.0.1-2PIGSTY.el9.aarch64.rpm
pg_tiktoken_16 0.0.1 el10.x86_64 pigsty 1.7 MiB pg_tiktoken_16-0.0.1-2PIGSTY.el10.x86_64.rpm
pg_tiktoken_16 0.0.1 el10.aarch64 pigsty 1.5 MiB pg_tiktoken_16-0.0.1-2PIGSTY.el10.aarch64.rpm
postgresql-16-pg-tiktoken 0.0.1 d12.x86_64 pigsty 1.4 MiB postgresql-16-pg-tiktoken_0.0.1-1PIGSTY~bookworm_amd64.deb
postgresql-16-pg-tiktoken 0.0.1 d12.aarch64 pigsty 1.2 MiB postgresql-16-pg-tiktoken_0.0.1-1PIGSTY~bookworm_arm64.deb
postgresql-16-pg-tiktoken 0.0.1 d13.x86_64 pigsty 1.4 MiB postgresql-16-pg-tiktoken_0.0.1-1PIGSTY~trixie_amd64.deb
postgresql-16-pg-tiktoken 0.0.1 d13.aarch64 pigsty 1.2 MiB postgresql-16-pg-tiktoken_0.0.1-1PIGSTY~trixie_arm64.deb
postgresql-16-pg-tiktoken 0.0.1 u22.x86_64 pigsty 1.6 MiB postgresql-16-pg-tiktoken_0.0.1-1PIGSTY~jammy_amd64.deb
postgresql-16-pg-tiktoken 0.0.1 u22.aarch64 pigsty 1.4 MiB postgresql-16-pg-tiktoken_0.0.1-1PIGSTY~jammy_arm64.deb
postgresql-16-pg-tiktoken 0.0.1 u24.x86_64 pigsty 1.6 MiB postgresql-16-pg-tiktoken_0.0.1-1PIGSTY~noble_amd64.deb
postgresql-16-pg-tiktoken 0.0.1 u24.aarch64 pigsty 1.4 MiB postgresql-16-pg-tiktoken_0.0.1-1PIGSTY~noble_arm64.deb
Package Version OS ORG SIZE File URL
pg_tiktoken_15 0.0.1 el8.x86_64 pigsty 1.7 MiB pg_tiktoken_15-0.0.1-2PIGSTY.el8.x86_64.rpm
pg_tiktoken_15 0.0.1 el8.aarch64 pigsty 1.6 MiB pg_tiktoken_15-0.0.1-2PIGSTY.el8.aarch64.rpm
pg_tiktoken_15 0.0.1 el9.x86_64 pigsty 1.6 MiB pg_tiktoken_15-0.0.1-2PIGSTY.el9.x86_64.rpm
pg_tiktoken_15 0.0.1 el9.aarch64 pigsty 1.5 MiB pg_tiktoken_15-0.0.1-2PIGSTY.el9.aarch64.rpm
pg_tiktoken_15 0.0.1 el10.x86_64 pigsty 1.7 MiB pg_tiktoken_15-0.0.1-2PIGSTY.el10.x86_64.rpm
pg_tiktoken_15 0.0.1 el10.aarch64 pigsty 1.5 MiB pg_tiktoken_15-0.0.1-2PIGSTY.el10.aarch64.rpm
postgresql-15-pg-tiktoken 0.0.1 d12.x86_64 pigsty 1.4 MiB postgresql-15-pg-tiktoken_0.0.1-1PIGSTY~bookworm_amd64.deb
postgresql-15-pg-tiktoken 0.0.1 d12.aarch64 pigsty 1.2 MiB postgresql-15-pg-tiktoken_0.0.1-1PIGSTY~bookworm_arm64.deb
postgresql-15-pg-tiktoken 0.0.1 d13.x86_64 pigsty 1.4 MiB postgresql-15-pg-tiktoken_0.0.1-1PIGSTY~trixie_amd64.deb
postgresql-15-pg-tiktoken 0.0.1 d13.aarch64 pigsty 1.2 MiB postgresql-15-pg-tiktoken_0.0.1-1PIGSTY~trixie_arm64.deb
postgresql-15-pg-tiktoken 0.0.1 u22.x86_64 pigsty 1.6 MiB postgresql-15-pg-tiktoken_0.0.1-1PIGSTY~jammy_amd64.deb
postgresql-15-pg-tiktoken 0.0.1 u22.aarch64 pigsty 1.4 MiB postgresql-15-pg-tiktoken_0.0.1-1PIGSTY~jammy_arm64.deb
postgresql-15-pg-tiktoken 0.0.1 u24.x86_64 pigsty 1.6 MiB postgresql-15-pg-tiktoken_0.0.1-1PIGSTY~noble_amd64.deb
postgresql-15-pg-tiktoken 0.0.1 u24.aarch64 pigsty 1.4 MiB postgresql-15-pg-tiktoken_0.0.1-1PIGSTY~noble_arm64.deb
Package Version OS ORG SIZE File URL
pg_tiktoken_14 0.0.1 el8.x86_64 pigsty 1.7 MiB pg_tiktoken_14-0.0.1-2PIGSTY.el8.x86_64.rpm
pg_tiktoken_14 0.0.1 el8.aarch64 pigsty 1.6 MiB pg_tiktoken_14-0.0.1-2PIGSTY.el8.aarch64.rpm
pg_tiktoken_14 0.0.1 el9.x86_64 pigsty 1.6 MiB pg_tiktoken_14-0.0.1-2PIGSTY.el9.x86_64.rpm
pg_tiktoken_14 0.0.1 el9.aarch64 pigsty 1.5 MiB pg_tiktoken_14-0.0.1-2PIGSTY.el9.aarch64.rpm
pg_tiktoken_14 0.0.1 el10.x86_64 pigsty 1.7 MiB pg_tiktoken_14-0.0.1-2PIGSTY.el10.x86_64.rpm
pg_tiktoken_14 0.0.1 el10.aarch64 pigsty 1.5 MiB pg_tiktoken_14-0.0.1-2PIGSTY.el10.aarch64.rpm
postgresql-14-pg-tiktoken 0.0.1 d12.x86_64 pigsty 1.4 MiB postgresql-14-pg-tiktoken_0.0.1-1PIGSTY~bookworm_amd64.deb
postgresql-14-pg-tiktoken 0.0.1 d12.aarch64 pigsty 1.2 MiB postgresql-14-pg-tiktoken_0.0.1-1PIGSTY~bookworm_arm64.deb
postgresql-14-pg-tiktoken 0.0.1 d13.x86_64 pigsty 1.4 MiB postgresql-14-pg-tiktoken_0.0.1-1PIGSTY~trixie_amd64.deb
postgresql-14-pg-tiktoken 0.0.1 d13.aarch64 pigsty 1.2 MiB postgresql-14-pg-tiktoken_0.0.1-1PIGSTY~trixie_arm64.deb
postgresql-14-pg-tiktoken 0.0.1 u22.x86_64 pigsty 1.6 MiB postgresql-14-pg-tiktoken_0.0.1-1PIGSTY~jammy_amd64.deb
postgresql-14-pg-tiktoken 0.0.1 u22.aarch64 pigsty 1.4 MiB postgresql-14-pg-tiktoken_0.0.1-1PIGSTY~jammy_arm64.deb
postgresql-14-pg-tiktoken 0.0.1 u24.x86_64 pigsty 1.6 MiB postgresql-14-pg-tiktoken_0.0.1-1PIGSTY~noble_amd64.deb
postgresql-14-pg-tiktoken 0.0.1 u24.aarch64 pigsty 1.4 MiB postgresql-14-pg-tiktoken_0.0.1-1PIGSTY~noble_arm64.deb
Package Version OS ORG SIZE File URL
pg_tiktoken_13 0.0.1 el8.x86_64 pigsty 1.7 MiB pg_tiktoken_13-0.0.1-2PIGSTY.el8.x86_64.rpm
pg_tiktoken_13 0.0.1 el8.aarch64 pigsty 1.6 MiB pg_tiktoken_13-0.0.1-2PIGSTY.el8.aarch64.rpm
pg_tiktoken_13 0.0.1 el9.x86_64 pigsty 1.6 MiB pg_tiktoken_13-0.0.1-2PIGSTY.el9.x86_64.rpm
pg_tiktoken_13 0.0.1 el9.aarch64 pigsty 1.5 MiB pg_tiktoken_13-0.0.1-2PIGSTY.el9.aarch64.rpm
pg_tiktoken_13 0.0.1 el10.x86_64 pigsty 1.7 MiB pg_tiktoken_13-0.0.1-2PIGSTY.el10.x86_64.rpm
pg_tiktoken_13 0.0.1 el10.aarch64 pigsty 1.5 MiB pg_tiktoken_13-0.0.1-2PIGSTY.el10.aarch64.rpm
postgresql-13-pg-tiktoken 0.0.1 d12.x86_64 pigsty 1.4 MiB postgresql-13-pg-tiktoken_0.0.1-1PIGSTY~bookworm_amd64.deb
postgresql-13-pg-tiktoken 0.0.1 d12.aarch64 pigsty 1.2 MiB postgresql-13-pg-tiktoken_0.0.1-1PIGSTY~bookworm_arm64.deb
postgresql-13-pg-tiktoken 0.0.1 d13.x86_64 pigsty 1.4 MiB postgresql-13-pg-tiktoken_0.0.1-1PIGSTY~trixie_amd64.deb
postgresql-13-pg-tiktoken 0.0.1 d13.aarch64 pigsty 1.2 MiB postgresql-13-pg-tiktoken_0.0.1-1PIGSTY~trixie_arm64.deb
postgresql-13-pg-tiktoken 0.0.1 u22.x86_64 pigsty 1.6 MiB postgresql-13-pg-tiktoken_0.0.1-1PIGSTY~jammy_amd64.deb
postgresql-13-pg-tiktoken 0.0.1 u22.aarch64 pigsty 1.4 MiB postgresql-13-pg-tiktoken_0.0.1-1PIGSTY~jammy_arm64.deb
postgresql-13-pg-tiktoken 0.0.1 u24.x86_64 pigsty 1.6 MiB postgresql-13-pg-tiktoken_0.0.1-1PIGSTY~noble_amd64.deb
postgresql-13-pg-tiktoken 0.0.1 u24.aarch64 pigsty 1.4 MiB postgresql-13-pg-tiktoken_0.0.1-1PIGSTY~noble_arm64.deb

Source

Repository
github.com/kelvich/pg_tiktoken
 Source Tarball
pg_tiktoken-0.0.1.tar.gz
 
pig build pkg pg_tiktoken;		# build rpm / deb with pig

Install

Make sure PGDG and PIGSTY repo available:

pig repo add pgdg pigsty -u   # add both repo and update cache

Install this extension with pig:

pig install pg_tiktoken;		# install via package name, for the active PG version

pig install pg_tiktoken -v 18;   # install for PG 18
pig install pg_tiktoken -v 17;   # install for PG 17
pig install pg_tiktoken -v 16;   # install for PG 16
pig install pg_tiktoken -v 15;   # install for PG 15
pig install pg_tiktoken -v 14;   # install for PG 14
pig install pg_tiktoken -v 13;   # install for PG 13

Create this extension with:

CREATE EXTENSION pg_tiktoken;
Last updated on
pg_summarizepg4ml