# Cohere Embedding Models
Source: https://thenile.dev/docs/ai-embeddings/embedding_models/cohere
## Availble models
Cohere has multiple models with various sizes and languages, here's the one we use in our example.
You can replace it with any embedding model supported by Cohere:
| Model | Dimensions | Max Tokens | Cost | MTEB Avg Score | Similarity Metrics |
| ------------------ | ---------- | ---------- | ------------------ | -------------- | ------------------ |
| embed-english-v3.0 | 1024 | 512 | \$0.10 / 1M Tokens | 64.5 | cosine |
## Usage
Cohere's API includes an `inputType` that allows you to specify the type of input you are embedding.
For example, is it a document that you will later retrieve? a question that you use for searching documents?
Texts for classification or clustering?
### Installing dependencies
```bash theme={null}
npm install @niledatabase/server cohere-ai
```
### Generating embeddings with Cohere
```javascript theme={null}
const { CohereClient } = require('cohere-ai');
const COHERE_API_KEY = 'your cohere api key';
const cohere = new CohereClient({
token: COHERE_API_KEY,
});
const model = 'embed-english-v3.0'; // replace with your favorite cohere model
const input_text = `The future belongs to those who believe in the beauty of their
dreams.`;
const doc_vec_resp = await cohere.embed({
texts: [input_text], // you can pass multiple texts to embed in one batch call
model: model,
inputType: 'search_document', // for embedding of documents stored in pg_vector
});
const question = 'Who does the future belong to?';
const question_vec_resp = await cohere.embed({
texts: [question],
model: model,
inputType: 'search_query', // for embedding of questions used to search documents
});
// Cohere's response is an object with an array of vector embeddings
// The object has other useful info like the model used, input text, billing, etc.
const doc_vec = doc_vec_resp.embeddings[0];
const question_vec = question_vec_resp.embeddings[0];
```
### Storing and retrieving the embeddings
```javascript theme={null}
// set up Nile as the vector store
const { Nile } = await import('@niledatabase/server');
const NILEDB_USER = 'you need a nile user with access to a nile database';
const NILEDB_PASSWORD = 'and a password for that user';
const nile = Nile({
user: NILEDB_USER,
password: NILEDB_PASSWORD,
});
// store vector in a table
await nile.db.query('INSERT INTO embeddings (embedding) values ($1)', [
JSON.stringify(doc_vec.map((v) => Number(v))),
]);
// search for similar vectors
let db_resp = await nile.db.query(
'select embedding from embeddings order by embedding<=>$1 limit 1',
[JSON.stringify(question_vec.map((v) => Number(v)))],
);
// Postgres returns an object, with array of rows each column is a
// property of the row the vector is represented as a string
let similar_str = db_resp.rows[0].embedding;
let similar = similar_str
.substring(1, similar_str.length - 1)
.split(',')
.map((v) => parseFloat(v));
// check that we got the same vector back
let same = true;
for (let i = 0; i < similar.length; i++) {
if (Math.abs(similar[i] - doc_vec[i]) > 0.000001) {
same = false;
}
}
console.log('got same vector? ' + same);
```
# Embedding Models served by DeepInfra
Source: https://thenile.dev/docs/ai-embeddings/embedding_models/deepinfra
## Availble models
Fireworks provide a larger variety of models, here's the one we use in our example. You can replace it with any
embedding model supported by DeepInfra:
| Model | Dimensions | Max Tokens | Cost | MTEB Avg Score | Similarity Metric |
| ------------------ | ---------- | ---------- | ------------------- | -------------- | ----------------- |
| thenlper/gte-large | 1024 | 512 | \$0.010 / 1M tokens | 63.23 | cosine |
## Usage
Note that DeepInfra doesn't have an SDK. Their documentation shows the use of the REST API directly with a
client library in your language of choice, or you can use OpenAI's SDK - DeepInfra is compatible with OpenAI's API.
In the examples below, we use OpenAI's SDK with DeepInfra URL, API key and models.
### Installing dependencies
```bash theme={null}
npm install @niledatabase/server openai
```
### Generating embeddings with DeepInfra
```javascript theme={null}
// set up OpenAI SDK with Fireworks configuration
const { OpenAI } = await import('openai');
const DEEPINFRA_API_KEY = 'your deepinfra api key';
const openai = new OpenAI({
apiKey: DEEPINFRA_API_KEY,
baseURL: 'https://api.deepinfra.com/v1/openai',
});
const MODEL = 'thenlper/gte-large'; // or any other model supported by DeepInfra
const input_text = `The future belongs to those who believe in the beauty of their
dreams.`;
// generate embeddings
let resp = await openai.embeddings.create({
model: MODEL,
input: input_text,
});
// OpenAI's response is an object with an array of
// objects that contain the vector embeddings
let embedding = resp.data[0].embedding;
```
### Storing and retrieving the embeddings
```javascript theme={null}
// set up Nile as the vector store
const { Nile } = await import('@niledatabase/server');
const NILEDB_USER = 'you need a nile user with access to a nile database';
const NILEDB_PASSWORD = 'and a password for that user';
const nile = Nile({
user: NILEDB_USER,
password: NILEDB_PASSWORD,
});
// store vector in a table
await nile.db.query('INSERT INTO embeddings (embedding) values ($1)', [
JSON.stringify(embedding.map((v) => Number(v))),
]);
// search for similar vectors
let db_resp = await nile.db.query(
'select embedding from embeddings order by embedding<=>$1 limit 1',
[JSON.stringify(embedding.map((v) => Number(v)))],
);
// Postgres returns an object, with array of rows each column is a
// property of the row the vector is represented as a string
let similar_str = db_resp.rows[0].embedding;
let similar = similar_str
.substring(1, similar_str.length - 1)
.split(',')
.map((v) => parseFloat(v));
// check that we got the same vector back
let same = true;
for (let i = 0; i < similar.length; i++) {
if (Math.abs(similar[i] - embedding[i]) > 0.000001) {
same = false;
}
}
console.log('got same vector? ' + same);
```
## Additional information
### Reducing dimensions
Using larger embeddings generally costs more and consumes more compute, memory and storage than using smaller embeddings. This is especially true for embeddings stored with `pg_vector`.
When storing embeddings in Postgres, it is important that each vector will be stored in a row that fits in a single PG block (typically 8K). If this size is exceeded,
the vector will be stored in TOAST storage which can slow down queries. In addition vectors that are "TOASTed" are not indexed, which means you can't reliably use vector indexes.
Fireworks supports multiple models. `gte-large` and `nomic-embed-text-v1.5` are two of the models available.
The `gte-large` model has 1024 dimensions and does not support scaling down. The `nomic-embed-text-v1.5` model has
768 dimensions and can scale down to 512, 256, 128 and 64.
# Embedding Models served by Fireworks
Source: https://thenile.dev/docs/ai-embeddings/embedding_models/fireworks
## Available models
Fireworks provide a larger variety of models, here are two of them. You can use any of the models supported by
Fireworks in the examples below:
| Model | Dimensions | Max Tokens | Cost | MTEB Avg Score | Similarity Metric |
| ------------------------------ | ----------------- | ---------- | ------------------- | -------------- | ----------------- |
| thenlper/gte-large | 1024 | 512 | \$0.016 / 1M tokens | 63.23 | cosine |
| nomic-ai/nomic-embed-text-v1.5 | 768 (scales down) | 8192 | \$0.008 / 1M tokens | 62.28 | cosine |
## Usage
Note that Fireworks doesn't have an SDK. Their documentation shows the use of the REST API directly with a
client library in your language of choice, or you can use OpenAI's SDK - Fireworks is compatible with OpenAI's API.
In the examples below, we use OpenAI's SDK with Fireworks URL, API key and models.
### Installing dependencies
```bash theme={null}
npm install @niledatabase/server openai
```
### Generating embeddings with Fireworks
```javascript theme={null}
// set up OpenAI SDK with Fireworks configuration
const { OpenAI } = await import('openai');
const FIREWORKS_API_KEY = 'your fireworks api key';
const openai = new OpenAI({
apiKey: FIREWORKS_API_KEY,
baseURL: 'https://api.fireworks.ai/inference/v1',
});
const MODEL = 'thenlper/gte-large'; // or nomic-ai/nomic-embed-text-v1.5
const input_text = `The future belongs to those who believe in the beauty of their
dreams.`;
// generate embeddings
let resp = await openai.embeddings.create({
model: MODEL,
input: input_text,
// nomic-ai/nomic-embed-text-v1.5 can scale down, thenlper/gte-large can't
// dimensions: 512,
});
// OpenAI's response is an object with an array of
// objects that contain the vector embeddings
let embedding = resp.data[0].embedding;
```
### Storing and retrieving the embeddings
```javascript theme={null}
// set up Nile as the vector store
const { Nile } = await import('@niledatabase/server');
const NILEDB_USER = 'you need a nile user with access to a nile database';
const NILEDB_PASSWORD = 'and a password for that user';
const nile = Nile({
user: NILEDB_USER,
password: NILEDB_PASSWORD,
});
// store vector in a table
await nile.db.query('INSERT INTO embeddings (embedding) values ($1)', [
JSON.stringify(embedding.map((v) => Number(v))),
]);
// search for similar vectors
let db_resp = await nile.db.query(
'select embedding from embeddings order by embedding<=>$1 limit 1',
[JSON.stringify(embedding.map((v) => Number(v)))],
);
// Postgres returns an object, with array of rows each column is a
// property of the row the vector is represented as a string
let similar_str = db_resp.rows[0].embedding;
let similar = similar_str
.substring(1, similar_str.length - 1)
.split(',')
.map((v) => parseFloat(v));
// check that we got the same vector back
let same = true;
for (let i = 0; i < similar.length; i++) {
if (Math.abs(similar[i] - embedding[i]) > 0.000001) {
same = false;
}
}
console.log('got same vector? ' + same);
```
## Additional information
### Reducing dimensions
Using larger embeddings generally costs more and consumes more compute, memory and storage than using smaller embeddings. This is especially true for embeddings stored with `pg_vector`.
When storing embeddings in Postgres, it is important that each vector will be stored in a row that fits in a single PG block (typically 8K). If this size is exceeded,
the vector will be stored in TOAST storage which can slow down queries. In addition vectors that are "TOASTed" are not indexed, which means you can't reliably use vector indexes.
Fireworks supports multiple models. `gte-large` and `nomic-embed-text-v1.5` are two of the models available.
The `gte-large` model has 1024 dimensions and does not support scaling down. The `nomic-embed-text-v1.5` model has
768 dimensions and can scale down to 512, 256, 128 and 64.
# Google Embedding Models
Source: https://thenile.dev/docs/ai-embeddings/embedding_models/google
Note that Google has to AI API products:
* [Google Cloud AI](https://cloud.google.com/ai-platform), also known as Vertex AI.
* [Gemini API](https://ai.google.dev/gemini-api/docs), also known as Generative Language API.
The NodeJS SDK for Gemini API is significantly nicer, so the example below will use it.
However, Vertex AI has more embedding models - multi-lingual, multi-modal, images and video.
## Availble models
The example below is based on the new `text-embedding-preview-0409` model, also called `text-embedding-0004` in Gemini APIs.\
It is the best text embedding model available from Google and ranks well in the MTEB benchmark.
| Model | Dimensions | Max Tokens | Cost | MTEB Avg Score | Similarity Metric |
| ------------------------------------------------- | ----------------- | ---------- | ------------------------------------------- | -------------- | ----------------- |
| text-embedding-preview-0409 / text-embedding-0004 | 768 (scales down) | 2048 | \$0.025/1M tokens in Vertex, free in Gemini | 66.31 | cosine, L2 |
## Usage
To use Google's embedding models, you need a Google Cloud project. The example below uses Gemini, so you will need to have
[Gemini Generative Language APIs enabled](https://console.developers.google.com/apis/api/generativelanguage.googleapis.com/overview).
and you will also need an API key with permissions to access the Generative Language API. You can get one by going to
[APIs & Services -> Credentials](https://console.cloud.google.com/apis/credentials) in your Google Cloud
Console. (You can also use Google's AI Studio to get an API key).
Vertex has separate API to enable, separate key permissions, separate pricing and a different SDK
(which we don't document here).
### Installing dependencies
```bash theme={null}
npm install @niledatabase/server @google/generative-ai
```
### Generating embeddings with Google
```javascript theme={null}
const { GoogleGenerativeAI } = require('@google/generative-ai');
const GOOGLE_API_KEY = 'my-google-api-key';
const model = 'models/text-embedding-004'; // or your favorite Google model
const genAI = new GoogleGenerativeAI(GOOGLE_API_KEY);
const embed = genAI.getGenerativeModel({
model: model,
outputDimensionality: 768, // optional, default 768 but can be scaled down
});
const input_text = `The future belongs to those who believe in the beauty of their
dreams.`;
// Note, if you don't want to specify taskType and title, you can simply use:
// const doc_vec_resp = await model.embedContent(input_text);
const doc_vec_resp = await embed.embedContent({
content: { parts: [{ text: input_text }] },
taskType: 'RETRIEVAL_DOCUMENT', // for embeddings of documents stored in pg_vector
// title of the document. can be used with RETRIEVAL_DOCUMENT and
// possibly improve the quality of the embeddings
// title: ""
});
const question = 'Who does the future belong to?';
const question_vec_resp = await embed.embedContent({
content: { parts: [{ text: question }] },
taskType: 'RETRIEVAL_QUERY', //embeddings of questions used to search documents
});
// Google returns a response object with an embedding field that
// contains the embeddings in the value field
const doc_vec = doc_vec_resp.embedding.values;
const question_vec = question_vec_resp.embedding.values;
```
### Storing and retrieving the embeddings
```javascript theme={null}
// set up Nile as the vector store
const { Nile } = await import('@niledatabase/server');
const NILEDB_USER = 'you need a nile user with access to a nile database';
const NILEDB_PASSWORD = 'and a password for that user';
const nile = Nile({
user: NILEDB_USER,
password: NILEDB_PASSWORD,
});
// create table to store vectors - vector size must match the model dimensions
await nile.db.query(
'CREATE TABLE IF NOT EXISTS embeddings (embedding vector(768))',
);
// store vector in a table
await nile.db.query('INSERT INTO embeddings (embedding) values ($1)', [
JSON.stringify(doc_vec.map((v) => Number(v))),
]);
// search for similar vectors
let db_resp = await nile.db.query(
'select embedding from embeddings order by embedding<=>$1 limit 1',
[JSON.stringify(question_vec.map((v) => Number(v)))],
);
// Postgres returns an object, with array of rows each column is a
// property of the row the vector is represented as a string
let similar_str = db_resp.rows[0].embedding;
let similar = similar_str
.substring(1, similar_str.length - 1)
.split(',')
.map((v) => parseFloat(v));
// check that we got the same vector back
let same = true;
for (let i = 0; i < similar.length; i++) {
if (Math.abs(similar[i] - doc_vec[i]) > 0.000001) {
same = false;
}
}
console.log('got same vector? ' + same);
```
## Additional notes
### Scale down
Google's `text-embedding-0004` model has 768 dimensions, but you can scale it down to lower dimensions.
The older model, `text-embedding-0001` does not support scaling down.
### Task types
Google's documentation about taskTypes is a bit confusing. Some documents say that `taskType` is only supported
by `text-embedding-0001` model, and other say that it works with `0004` as well. My experiments showed that
`taskType` works with `0004`, so I have included it in the example above. I assume there are typos in the docs.
### Distance metrics
Google documentation doesn't mention the distance metric used for similarity search and doesn't mention anything
about normalization either. However, the MTEB benchmark records for Google's model show use of Cosine and L2 distance.
# Open AI Embedding Models
Source: https://thenile.dev/docs/ai-embeddings/embedding_models/open_ai
## Availble models
| Model | Dimensions | Max Tokens | Cost | MTEB Avg Score | Similarity Metric |
| ---------------------- | ------------------ | ---------- | ------------------ | -------------- | ----------------------- |
| text-embedding-3-small | 1536 (scales down) | 8191 | \$0.02 / 1M tokens | 62.3 | cosine, dot product, L2 |
| text-embedding-3-large | 3072 (scales down) | 8191 | \$0.13 / 1M tokens | 64.6 | cosine, dot product, L2 |
## Usage
### Installing dependencies
```bash theme={null}
npm install @niledatabase/server openai
```
### Generating embeddings with OpenAI
```javascript theme={null}
// set up OpenAI
const { OpenAI } = await import('openai');
const OPENAI_API_KEY = 'your openai api key';
const openai = new OpenAI({
apiKey: OPENAI_API_KEY, // not necessary if you have this env variable
});
const MODEL = 'text-embedding-3-small'; // or text-embedding-3-large
const input_text = `The future belongs to those who believe in the beauty of their
dreams.`;
// generate embeddings
let resp = await openai.embeddings.create({
model: MODEL,
input: input_text,
dimensions: 1024, // OpenAI models scale down to lower dimensions
});
// OpenAI's response is an object with an array of
// objects that contain the vector embeddings
let embedding = resp.data[0].embedding;
```
### Storing and retrieving the embeddings
```javascript theme={null}
// set up Nile as the vector store
const { Nile } = await import('@niledatabase/server');
const NILEDB_USER = 'you need a nile user with access to a nile database';
const NILEDB_PASSWORD = 'and a password for that user';
const nile = Nile({
user: NILEDB_USER,
password: NILEDB_PASSWORD,
});
// store vector in a table
await nile.db.query('INSERT INTO embeddings (embedding) values ($1)', [
JSON.stringify(embedding.map((v) => Number(v))),
]);
// search for similar vectors
let db_resp = await nile.db.query(
'select embedding from embeddings order by embedding<=>$1 limit 1',
[JSON.stringify(embedding.map((v) => Number(v)))],
);
// Postgres returns an object, with array of rows each column is a
// property of the row the vector is represented as a string
let similar_str = db_resp.rows[0].embedding;
let similar = similar_str
.substring(1, similar_str.length - 1)
.split(',')
.map((v) => parseFloat(v));
// check that we got the same vector back
let same = true;
for (let i = 0; i < similar.length; i++) {
if (Math.abs(similar[i] - embedding[i]) > 0.000001) {
same = false;
}
}
console.log('got same vector? ' + same);
```
## Additional information
### Reducing dimensions
Using larger embeddings generally costs more and consumes more compute, memory and storage than using smaller embeddings. This is especially true for embeddings stored with `pg_vector`.
When storing embeddings in Postgres, it is important that each vector will be stored in a row that fits in a single PG block (typically 8K). If this size is exceeded,
the vector will be stored in TOAST storage which can slow down queries. In addition vectors that are "TOASTed" are not indexed, which means you can't reliably use vector indexes.
Both OpenAI's embedding models were trained with a technique that allows developers to trade-off performance and cost of using embeddings. Specifically, developers can shorten embeddings
(i.e. remove some numbers from the end of the sequence) without the embedding losing its concept-representing properties by passing in the dimensions API parameter.
Using the dimensions parameter when creating the embedding (as shown above) is the preferred approach. In certain cases, you may need to change the embedding dimension after you generate it.
When you change the dimension manually, you need to be sure to normalize the dimensions of the embedding.
### Distance metrics
OpenAI embeddings are normalized to length 1, which means that you can use L2, cosine, and dot product similarity metrics interchangeably. Dot product is the fastest to compute.
# Embedding Models for RAG
Source: https://thenile.dev/docs/ai-embeddings/embedding_models/overview
Embedding models are used to convert text into vector embeddings.
These embeddings can be used to perform various tasks like similarity search, clustering, and classification.
In the context of RAG, embedding models are used to convert the input text into embeddings that are used to retrieve relevant (similar)
documents from the document store.
Vendor(s)
Model
dimensions
max tokens
cost
MTEB avg score
similarity metric
OpenAI
text-embedding-3-small
1536 (scales down)
8191
\$0.02 / 1M tokens
62.3
cosine, dot product, L2
text-embedding-3-large
3072 (scales down)
8191
\$0.13 / 1M tokens
64.6
cosine, dot product, L2
Google
text-embedding-preview-0409 / text-embedding-0004
768 (scales down)
2048
\$0.025/1M tokens in Vertex, free in Gemini
66.31
cosine, L2
Fireworks
thenlper/gte-large
1024
512
\$0.016 / 1M tokens
63.23
cosine
nomic-ai/nomic-embed-text-v1.5
768 (scales down)
8192
\$0.008 / 1M tokens
62.28
cosine
DeepInfra
gte-large
1024
512
\$0.010 / 1M tokens
63.23
cosine
Cohere
embed-english-v3.0
1024
512
\$0.10 / 1M Tokens
64.5
cosine
Voyage
voyage-large-2-instruct
1024
16000
\$0.12 / 1M tokens
68.28
cosine, dot product, L2
voyage-2
1024
4000
\$0.1/ 1M tokens
cosine, dot product, L2
voyage-code-2
1536
16000
\$0.12/ 1M tokens
cosine, dot product, L2
voyage-law-2
1024
16000
\$0.12/ 1M tokens
cosine, dot product, L2
## Explanation of columns
* **Vendor(s)**: The vendor(s) that provide the model as a service.
* **Model**: The name of the model.
* **dimensions**: The number of dimensions in the vector embeddings that the model generates
* **max tokens**: The maximum number of tokens that can be passed to the model in a single request
* **cost**: The cost of using the model (based on vendor pricing page, where available)
* **MTEB avg score**: The [Massive Text Embedding Benchmark (MTEB)](https://github.com/embeddings-benchmark/mteb) average score. MTEB is a benchmark for evaluating the quality of embeddings across a range of tasks. The higher the score, the better the embeddings.
* **similarity metric**: The similarity metric recommended by the model authors to use with the embeddings. We only included the metrics supported by `pg_vector`, some of the models may support additional metrics.
# Voyage Embedding Models
Source: https://thenile.dev/docs/ai-embeddings/embedding_models/voyage
## Availble models
Voyage has a collection of specialized models for embedding text from different domains: financial, legal
(and large documents), code and medical.
It also has highly ranked general embedding model that can be used for a variety of tasks,
a general model that is optimized for retrieval, and a smaller cost-efficient retrieval model.
We included a subset here for reference:
| Model | Dimensions | Max Tokens | Cost | MTEB Avg Score | Similarity Metric |
| ----------------------- | ---------- | ---------- | ------------------ | -------------- | ----------------------- |
| voyage-large-2-instruct | 1024 | 16000 | \$0.12 / 1M tokens | 68.28 | cosine, dot product, L2 |
| voyage-2 | 1024 | 4000 | \$0.1 / 1M tokens | | cosine, dot product, L2 |
| voyage-code-2 | 1536 | 16000 | \$0.12 / 1M tokens | | cosine, dot product, L2 |
| voyage-law-2 | 1024 | 16000 | \$0.12 / 1M tokens | | cosine, dot product, L2 |
## Usage
Voyage has a Python, but not a Javascript, SDK. Their REST API is **almost** compatible with OpenAI's API,
but unfortunately, their powerful general purpose model, `voyage-large-2-instruct` requires `inputType` parameter,
which is not supported by OpenAI's SDK. Fortunately, LangChain has a nice community-contributed JS library for Voyage,
which supports the `inputType` parameter. So we are going to use the LangChain community library in the example below.
### Installing dependencies
```bash theme={null}
npm install @niledatabase/server @langchain/community
```
### Generating embeddings with Voyage
```javascript theme={null}
const VOYAGE_API_KEY = 'your voyage api key';
const { VoyageEmbeddings } =
await import('@langchain/community/embeddings/voyage');
// we need a separate model object for documents and queries
// because the inputType is different and is in this object
const embedDocs = new VoyageEmbeddings({
apiKey: VOYAGE_API_KEY, // In Node.js defaults to process.env.VOYAGEAI_API_KEY
inputType: 'document', // for the embedding of documents stored in pg_vector
});
const embedQueries = new VoyageEmbeddings({
apiKey: VOYAGE_API_KEY,
inputType: 'query', // for the embedding of questions used to search documents
});
const input_text = `The future belongs to those who believe in the beauty of their
dreams.`;
// embedDocuments is the batch API and can take multiple documents in one call
const doc_vec_resp = await embedDocs.embedDocuments([input_text]);
// it returns an array of embeddings, we have just one
const doc_vec = doc_vec_resp[0];
const question = 'Who does the future belong to?';
// embedQuery takes a single query string and returns a single vector
const question_vec = await embedQueries.embedQuery(question);
```
### Storing and retrieving the embeddings
```javascript theme={null}
// set up Nile as the vector store
const { Nile } = await import('@niledatabase/server');
const NILEDB_USER = 'you need a nile user with access to a nile database';
const NILEDB_PASSWORD = 'and a password for that user';
const nile = Nile({
user: NILEDB_USER,
password: NILEDB_PASSWORD,
});
// store vector in a table
await nile.db.query('INSERT INTO embeddings (embedding) values ($1)', [
JSON.stringify(doc_vec.map((v) => Number(v))),
]);
// search for similar vectors
let db_resp = await nile.db.query(
'select embedding from embeddings order by embedding<=>$1 limit 1',
[JSON.stringify(question_vec.map((v) => Number(v)))],
);
// Postgres returns an object, with array of rows each column is a
// property of the row the vector is represented as a string
let similar_str = db_resp.rows[0].embedding;
let similar = similar_str
.substring(1, similar_str.length - 1)
.split(',')
.map((v) => parseFloat(v));
// check that we got the same vector back
let same = true;
for (let i = 0; i < similar.length; i++) {
if (Math.abs(similar[i] - doc_vec[i]) > 0.000001) {
same = false;
}
}
console.log('got same vector? ' + same);
```
## Additional information
### Distance metrics
Voyage embeddings are normalized to length 1, which means that you can use L2, cosine, and dot product similarity
metrics interchangeably. Dot product is the fastest to compute.
### API Rate limits
Note that on the free plan, the rate limits are quite strict. Voyage gives you only 3 API calls a minute.
Which means that after you embedded some documents, you can only generate embeddings for 2 queries before
running out of the minute and having to wait.
# Generative AI & Embeddings
Source: https://thenile.dev/docs/ai-embeddings/introduction
Generative AI is truly transforming every industry and vertical in B2B. It significantly improves the experience of the product, the value the user receives and increases the overall productivity.
For example,
1. A corporate wiki (eg. Confluence, Notion) where the employees can perform semantic search on their companies data
2. A chatbot for a CRM (eg. Salesforce, Hubspot) that sales reps can use to ask questions about past and future customer deals and can have a back and forth conversation
3. An autopilot for developers in their code repository (Github, Gitlab) to improve productivity. The autopilot should run on the companies code as well apart from learning on public repositories.
## Challenges with AI in B2B
*Separate databases for vector embeddings and customer data*
In recent years, numerous vector databases have emerged. This trend separates customers' core data and metadata from their embeddings, forcing companies to manage multiple databases. Such separation increases costs, significantly complicates application development and operation, and leads to inefficient resource utilization between vector embeddings and customer metadata. Moreover, keeping these databases synchronized with customer changes adds yet another layer of complexity.
*Lack of isolation for customer workloads*
AI workloads demand significantly more memory and compute than traditional SaaS workloads. Customer adoption and growth are much faster with AI, though some of this can be attributed to a hype cycle. Moreover, rebuilding indexes for embeddings requires additional resources and may impact production workloads. The ability to isolate customer data and their AI workloads has a significant impact on the customer's experience. Isolation is a key customer requirement (no one wants their data mixed with anyone else’s) and also critical to performance - 3 million embeddings is very large. 1000 tenants with 3000 embeddings each is very manageable - you get lower latency and 100% recall.
*Scaling to billions of embeddings across customers*
AI workloads scale to 50-100 million embeddings and in some cases even a billion embeddings. The biggest unlock with AI is the ability to search through unstructured data. All the data in different PDFs, Images, Wikis are now searchable. In addition, these unstructured data need to be chunked to do better contextual search. The explosion of vector embeddings requires a scalable database that can store billions of embeddings at a really low cost.
*Connecting all the customer’s data to the OLTP*
90% of AI use cases involve extracting data from customers' various SaaS services, making it accessible to LLMs, and allowing users to write prompts against this data. For instance, Glean, an AI-first company, aggregates data from issue trackers, wikis, and Salesforce, making it searchable in one central location using LLMs. Glean must offer a streamlined process for each customer to extract data from their SaaS APIs and transfer it to Glean's database. This data needs to be stored and managed on a per-customer basis. Vector embeddings must be computed during data ingestion. In the AI era, ETL pipelines from SaaS services to OLTP databases need to be reimagined for each customer.
*Cost of computing, storing and querying customer vector embeddings*
The sheer scale of vector embeddings and their associated workloads significantly increases the cost of managing AI infrastructure. The primary expenses stem from compute and storage, which typically align with customer activity. Ideally, you'd want to pay only for the exact resources a customer uses for compute. Similarly, you'd prefer cheaper storage options when embeddings aren't being accessed. By implementing per-customer cost management for their workloads, it should be possible to reduce expenses by 10 to 20 times.
## What are embeddings?
In generative AI development, embeddings refer to numerical representations of data that capture meaningful relationships, semantics, or context within the data. These representations are often used to convert high-dimensional, categorical, or unstructured data into lower-dimensional, continuous vectors that can be processed by machine learning models.
1. **Word Embeddings**
Word embeddings are one of the most common types of embeddings. They represent words from a vocabulary as dense numerical vectors in a lower-dimensional space. Word embeddings capture semantic and syntactic relationships between words. For example, words with similar meanings will have similar embeddings, and word arithmetic can be performed using embeddings (e.g., "king" - "man" + "woman" ≈ "queen"). Well-known word embedding methods include Word2Vec, GloVe, FastText, and BERT.
2. **Sentence and Document Embeddings**:
Instead of representing individual words, sentence and document embeddings represent entire sentences, paragraphs, or documents as numerical vectors.These embeddings aim to capture the overall meaning and context of the text. They are useful for applications like text summarization, document classification, and sentiment analysis. Models like BERT and the Universal Sentence Encoder can generate sentence and document embeddings.
3. **Image Embeddings**:
In computer vision, image embeddings represent images as vectors in a lower-dimensional space. Image embeddings capture visual features, allowing generative AI models to understand and generate images or perform tasks like image search and object detection. Convolutional Neural Networks (CNNs) are commonly used to generate image embeddings.
There are many ways to compare embeddings. L2 distance (Euclidean distance), inner product, and cosine distance are different similarity or dissimilarity measures used to compare vectors in multi-dimensional spaces.
Let's use sentence embeddings to explain how embeddings help with finding the similarity between sentences:
**Example Sentences**:
**Sentence 1**: "The sun rises in the east every morning."
**Sentence 2**: "The moon sets in the west at night."
**Sentence 3**: "Bananas are a source of potassium."
**Sentence Embeddings** (Hypothetical Values in a 4-dimensional space):
* Sentence 1 Embedding: \[2.2, 1.0, -0.8, 0.9]
* Sentence 2 Embedding: \[2.0, 1.3, 0.9, 1.1]
* Sentence 3 Embedding: \[0.6, 2.4, 2.1, 0.8]
**Similarity Calculation**:
We'll use cosine similarity to measure the similarity between sentence embeddings. The closer the cosine similarity value is to 1, the more similar the sentences are:
* Cosine Similarity between Sentence 1 and Sentence 2 ≈ 0.979
* Cosine Similarity between Sentence 1 and Sentence 3 ≈ 0.089
* Cosine Similarity between Sentence 2 and Sentence 3 ≈ 0.083
In this example, we used sentence embeddings to represent the entire sentences in a four-dimensional space. The cosine similarity between Sentence 1 and Sentence 2 is approximately 0.979, indicating high similarity because both sentences share a similar context related to celestial objects and directions. Sentence 3, which discusses a different topic, has lower similarity with both Sentence 1 and Sentence 2.
## Support for embeddings in Nile - pg\_vector
Embeddings in Nile is enabled using pg\_vector, the Postgres extension. This extension is enabled by default in Nile and is available to be used once you create a database. You can read more about how to use pg\_vector and build a real world AI native B2B application in the pg\_vector section.
# Model Context Protocol
Source: https://thenile.dev/docs/ai-embeddings/model_context_protocol
Model Context Protocol for Nile
The Model Context Protocol (MCP) is a standardized interface that enables Large Language Models (LLMs) like Claude to interact with external tools and services. For Nile, MCP provides a natural language interface to manage databases, execute queries, and handle database operations through AI assistants.
Key benefits:
* Natural language interaction with your databases
* Standardized interface across different AI tools
* Type-safe operations with input validation
* Real-time database management and querying
* Secure credential handling
## Installing Nile Model Context Protocol
### Stable Version
```bash theme={null}
npm install @niledatabase/nile-mcp-server
```
This will install @niledatabase/nile-mcp-server in your node\_modules folder. For example: node\_modules/@niledatabase/nile-mcp-server/dist/
### Latest Alpha/Preview Version
```bash theme={null}
npm install @niledatabase/nile-mcp-server@alpha
```
This will install @niledatabase/nile-mcp-server in your node\_modules folder. For example: node\_modules/@niledatabase/nile-mcp-server/dist/
### Manual Installation
```bash theme={null}
# Clone the repository
git clone https://github.com/yourusername/nile-mcp-server.git
cd nile-mcp-server
# Install dependencies
npm install
# Build the project
npm run build
```
### Configuration
Create a `.env` file in your project root:
```env theme={null}
NILE_API_KEY=your_api_key_here
NILE_WORKSPACE_SLUG=your_workspace_slug
```
You can get the API key and workspace from [Nile console](https://console.thenile.dev/). Note the workspace name at the top and you can create an API key from the Security tab on the left.
## Available Tools
### Database Management
* **create-database**: Create new databases
* **list-databases**: List all databases in workspace
* **get-database**: Get detailed database information
* **delete-database**: Delete databases
### Credential Management
* **list-credentials**: List database credentials
* **create-credential**: Create new database credentials
### Region Management
* **list-regions**: List available regions for database creation
### SQL Operations
* **execute-sql**: Execute SQL queries with results as markdown tables
### Resource Management
* **read-resource**: Get detailed schema information
* **list-resources**: List all database resources
### Tenant Management
* **list-tenants**: List all database tenants
* **create-tenant**: Create new tenants
* **delete-tenant**: Delete existing tenants
## Server Modes
### STDIN Mode (Default)
STDIN mode uses standard input/output for communication, ideal for direct integration with AI tools.
1. Start the server:
```bash theme={null}
node dist/index.js
```
2. The server automatically uses STDIN/STDOUT for communication
### SSE Mode
Server-Sent Events (SSE) mode enables HTTP-based, event-driven communication.
1. Configure SSE mode in `.env`:
```env theme={null}
MCP_SERVER_MODE=sse
```
2. Start the server:
```bash theme={null}
node dist/index.js
```
3. Connect to SSE endpoint:
```bash theme={null}
# Listen for events
curl -N http://localhost:3000/sse
```
4. Send commands:
```bash theme={null}
# Send a command
curl -X POST http://localhost:3000/messages \
-H "Content-Type: application/json" \
-d '{
"type": "function",
"name": "list-databases",
"parameters": {}
}'
```
## Claude Desktop Setup
1. Install [Claude Desktop](https://claude.ai/desktop)
2. Configure MCP Server:
* Open Claude Desktop
* Go to Settings > MCP Servers
* Click "Add Server"
* Add configuration:
```json theme={null}
{
"mcpServers": {
"nile-database": {
"command": "node",
"args": ["/path/to/your/nile-mcp-server/dist/index.js"],
"env": {
"NILE_API_KEY": "your_api_key_here",
"NILE_WORKSPACE_SLUG": "your_workspace_slug"
}
}
}
}
```
3. Replace:
* `/path/to/your/nile-mcp-server` with your actual project path
* `your_api_key_here` with your Nile API key
* `your_workspace_slug` with your workspace slug
4. Click Save and restart Claude Desktop
## Cursor Setup
1. Install [Cursor](https://cursor.sh)
2. Configure MCP Server:
* Open Cursor
* Go to Settings (⌘,) > Features > MCP Servers
* Click "Add New MCP Server"
3. Add server configuration:
* Name: `nile-database`
* Command:
```bash theme={null}
env NILE_API_KEY=your_key NILE_WORKSPACE_SLUG=your_workspace node /absolute/path/to/nile-mcp-server/dist/index.js
```
4. Replace:
* `your_key` with your Nile API key
* `your_workspace` with your workspace slug
* `/absolute/path/to` with your project path
5. Click Save and restart Cursor
## Windsurf Setup
1. Open Windsurf and go to the Cascade assistant
2. Click on the hammmer icon (should show MCP on hover) and then on configure
3. Paste the configuration below into the mcp\_configuration.json file
```json theme={null}
{
"mcpServers": {
"nile-database": {
"command": "node",
"args": ["/path/to/your/nile-mcp-server/dist/index.js"],
"env": {
"NILE_API_KEY": "your_api_key_here",
"NILE_WORKSPACE_SLUG": "your_workspace_slug"
}
}
}
}
```
4. Replace:
* `/path/to/your/nile-mcp-server` with your actual project path
* `your_api_key_here` with your Nile API key
* `your_workspace_slug` with your workspace slug
5. On saving, Windsurf will ask for confirmation. Once confirmed, it should show nile-database MCP active with a green active status
## Cline Setup
1. Click on MCP Servers in the Cline chat window
2. Click on the Installed tab and tap the "Configure MCP Servers" SignOutButton
3. Paste the configuration below into the cline\_mcp\_settings.json file
```json theme={null}
{
"mcpServers": {
"nile-database": {
"command": "node",
"args": ["/path/to/your/nile-mcp-server/dist/index.js"],
"env": {
"NILE_API_KEY": "your_api_key_here",
"NILE_WORKSPACE_SLUG": "your_workspace_slug"
}
}
}
}
```
4. Replace:
* `/path/to/your/nile-mcp-server` with your actual project path
* `your_api_key_here` with your Nile API key
* `your_workspace_slug` with your workspace slug
5. On saving, Cline will ask for confirmation. Once confirmed, it should show nile-database MCP active with a green active status
## Claude Code Setup
1. Install Claude Code
```bash theme={null}
npm install -g @anthropic-ai/claude-code
```
2. Add the MCP Server to Claude Code (note that right now Claude Code only supports STDIO mode, not SSE):
```bash theme={null}
claude mcp add nile_local_mcp node /path/to/your/nile-mcp-server/dist/index.js -e NILE_API_KEY=your_api_key_here -e NILE_WORKSPACE_SLUG=your_workspace_slug
```
3. You should see the following output:
```bash theme={null}
Added stdio MCP server nile_local_mcp with command: node /path/to/your/nile-mcp-server/dist/index.js to project config
```
4. You can now use the MCP Server in Claude Code:
## Example Usage
```bash theme={null}
# Create a database
"Create a new database named 'my-app' in AWS_US_WEST_2"
# List databases
"Show me all my databases"
# Execute SQL
"Run this query on my-app: SELECT * FROM users LIMIT 5"
# Get schema information
"Show me the schema for the users table in my-app"
```
## Troubleshooting
### Common Issues
1. **Connection Issues**
* Verify API key and workspace slug
* Check server path in configuration
* Ensure project is built (`npm run build`)
2. **Permission Issues**
* Verify API key permissions
* Check workspace access
* Ensure correct environment variables
3. **Tool Access Issues**
* Restart AI tool (Claude/Cursor)
* Check server logs for errors
* Verify tool configuration
### Getting Help
* Check server logs in the `logs` directory
* Review error messages in AI tool
* Ensure all environment variables are set
* Verify network connectivity
# Retrieval Augmented Generation (RAG)
Source: https://thenile.dev/docs/ai-embeddings/rag
RAG, or retrieval augumented generation is one of the most popular methods of building applications on top of general large language model. It allows using the knowledge and language skills of a world-class pre-trained model, while delivering results specific to a vertical or a company.
## Why RAG?
Generative AI models like GPT-3 are powerful, but they are not perfect. They can generate text that is incorrect, biased, or not relevant to the context.
After all, they are trained on information from the internet, and people often post wrong, misleading or fake information.
But the main limitation is that they do not have access to specific knowledge or data. ChatGPT can write an employee handbook for a company,
but it cannot answer questions about the company's specific policies or procedures.
There are two popular ways to address this core limitations, and they can be used separately or together. One is to **fine-tune** a model on a specific dataset or task.
And the other is **RAG**. For every question asked, retrieve the most relevant information from the preferred sources, and provide it to the generative model as context to use when responding to the question.
RAG is generally cheaper, faster and lower-effort than fine-tuning, and it can be used to improve results from any generative model, so it is a popular choice for many applications.
## RAG Architecture
From the short description above, you can see that the key to successful RAG is the ability to provide the generative model with the most relevant information.
And this requires preparing and storing data in a way that makes it efficient (and even plain possible) to retrieve the most relevant information.
Putting together the data preparation, storage, and retrieval methods with the generative model in order to have an informed conversation with a generative model is the essence of
retrieval-augmented generation (RAG). The general architecture of RAG-based application is illustrated in the figure below:
As you can see, RAG based applications typically have two independent phases:
* **Ingestion phase:** In which they load the source documents, split them into chunks, generate vector embeddings, store them in a database, and optionally generate an index on the stored vectors.
* **Conversation phase:** In which they take the user question, generate a vector embedding, search the stored vectors for related chunks, and then send the question together with the related text and perhaps older questions and responses in the conversation and a creative prompt to ChatGPT’s conversation API. When ChatGPT responds, the question and answer are stored and displayed to the user.
Lets discuss each of these tasks in more detail. We'll start with the last task, the retrieval of relevant information, since our choice of retrieval method will influence
the data preparation and storage methods.
### Retrieval Methods
For text retrieval, there are two popular methods, and a hybrid of the two:
**Full-text search**
Full-text search is the more traditional approach. Text search algorithms rely on word frequency (how often the word appears in each text vs in general) and
lexical similarity (The word “ice cream” is similar to “ice” and “cream” but not to “scoop” and “vanilla”). These algorithms can handle typos, synonyms,
partial words, and fuzzy matching.
**Vector search (also called semantic search)**
This method uses AI models, based on the transformer architecture, to find documents that are similar in meaning.
These specially trained transformers (called **embedding models** or embedders) convert text to a vector (also called **embedding**). The vector doesn't represent exactly the
words in the text but rather the semantic meaning of the text. The embedding model is trained on a giant corpus of texts (such as the entire internet), so it has “knowledge” of
which terms are related. Once texts have been converted to vectors, you can check if any two texts are related by checking how close the two vectors are. When
checking the similarity of texts using embeddings, it is expected that “ice cream” will be fairly close to “scoop” and “vanilla” since these words often show up
next to each other.
The popular embedders are based on transformer models, fine tuned for the task of generating embeddings. There are also specialized embedding models, pre-trained or fine-tuned for specific subsets of the language such as code,
legalese or medical jargon. Words like "function", "class" and "variable" will be close to each other in the vector space of a code embedding model, but not in a general english-language model.
**Hybrid**
In the hybrid approach, you use each method separately to find the closest matching text. Then, you combine the distance score from each method
(the combination is typically weighted, and the scores have to be normalized first). Then, you re-rank the results based on the combined score
and finally choose the highest-ranking documents based on the combined score.
There are also a few embedding models (notably, [BGE-M3](https://arxiv.org/pdf/2402.03216)) that combine the two methods in the same model, and
use relevancy scores from each model to re-rank the results themselves.
### Storing Documents and Embeddings
Regardless of the method you choose, you will need a data store that can efficiently store and retrieve the vectors or the text based on these methods. Some data
stores that specialize in one of the two algorithms, and a few offer both. Postgres has built-in full text search and extensions for both vector and text search.
[Pg\_vector](https://www.thenile.dev/docs/ai-embeddings/pg_vector), which is built into Nile, has an efficient implementation of vector *distance metrics*, so you can order results by distance
(also called ranking) or efficiently find vectors within a certain distance of another vector. There are many distance metrics, and different databases support different
ones, the most common ones are: **Cosine distance**, the most popular distance metric for text search, and **dot product** which is more efficient but only works for normalized vectors.
In addition, vector databases have indexes that can make searching large collections of vectors even more efficient. These indexes are different from
those you are familiar with because they are based on machine learning algorithms. These algorithms find close vectors very efficiently, even with millions
of vectors. But they have accuracy and memory use tradeoffs. Because of the accuracy tradeoffs (also known as **recall tradeoffs**, since it looks like the database
“forgot” some of the data), the algorithms used in the indexes are called **ANN - approximate nearest neighbors** (as opposed to **KNN** - the accurate results
you get by fully scanning the vector collection). Because these indexes use machine learning techniques, creating them is often called training.
Just don't get confused: this type of training fundamentally differs from the pre-training or fine-tuning of large language models.
You can read more about the [pg\_vector extension](https://www.thenile.dev/docs/ai-embeddings/pg_vector) and how to use it in the Nile documentation.
### Preparing documents for storage and embedding
Before you can generate embeddings or create text search indexes for documents, you need to prepare them. The preparation steps depend on the document type and the retrieval method you choose.
For example, if you use PDFs or HTML documents, you will need to extract the text from them. If you used scanned documents, you will need to use OCR to extract the text.
If you use full-text search you will need to tokenize the text, remove stop words and possibly chunk it into smaller portions. If you use embeddings, you will need to chunk the text, possibly normalize it and generate embeddings for each chunk.
The specific preparation steps you'll need also depend on the search or embedding model you use - size limits on the input, whether you need to pad short documents, how it handles common words, etc.
Check the model documentation for the specific requirements.
**Chunking** is the most common step in preparing documents. It is almost mandatory for two reasons:
* Embedding models have a limit on input text size, although this is improving and popular models can often accept 4K or even 8K tokens (words or subwords).
* Providing generative models with smaller chunks that have more closely related information usually works better than providing the model with larger, somewhat relevant chunks. [AI results degrade with large context windows](https://arxiv.org/abs/2307.03172).
## Simple RAG Example with OpenAI and Nile
In this example, we will use OpenAI's GPT-3.5 model and Nile's pg\_vector extension to demonstrate how RAG works.
We will embed a few documents, and then search for the most relevant document to a user's question:
```javascript theme={null}
const { OpenAIEmbeddings } = await import('@langchain/openai');
const { Nile } = await import('@niledatabase/server');
const EMBEDDING_MODEL = 'text-embedding-3-large';
const OPEN_API_KEY = 'bring your own key';
const NILEDB_USER = 'we use nile as the vector store, so we need a user';
const NILEDB_PASSWORD = 'and their password';
let model = new OpenAIEmbeddings({
apiKey: OPEN_API_KEY,
model: EMBEDDING_MODEL,
dimensions: 1024, // we'll explain why in the next blog
});
let nile = Nile({
user: NILEDB_USER,
password: NILEDB_PASSWORD,
});
// some documents:
const documents = [
'JavaScript is a programming language commonly used in web development.',
'Node.js is a runtime environment that allows you to run JavaScript on the server side.',
'React is a JavaScript library for building user interfaces.',
];
// embed documents
let vectors = await model.embedDocuments(documents);
// store embeddings
await nile.db.query(
'CREATE TABLE IF NOT EXISTS embeddings (id integer, embedding vector(1024));',
);
for (const [i, vec] of vectors.entries()) {
await nile.db.query(
'INSERT INTO embeddings (id, embedding) values ($1, $2)',
[i, JSON.stringify(vec.map((v) => Number(v)))],
);
}
// now lets ask a question
let question_vec = await model.embedDocuments(['Tell me about React']);
// search for the nearest document by cosine distance of embedding
let answer_vec = await nile.db.query(
'select id from embeddings order by embedding<=>$1 limit 1',
[JSON.stringify(question_vec[0])],
);
// return the answer:
console.log(
'based on your question, this document is relevant: ' +
documents[answer_vec.rows[0].id],
);
```
As you can see in the example, each step in RAG is simple and can be implemented in a single line of code - embedding, storing, and searching.
But the result is powerful and allows you to build AI-native SaaS applications that can answer questions about specific topics with greater accuracy, like a human expert would.
## RAG best practices
## Use cases for RAG
RAG performs best when:
* You have more source material than can fit in the generative model's context window.
* You have a large number of questions that can be answered by the source material. So you can embed and index the source material once and then use it to answer many questions.
* The source material is structured or can be chunked into smaller parts that are relevant to the questions.
Here are a few examples of use cases where RAG can be very useful:
* **Customer Support Automation:** RAG can be used to automate customer support by providing answers to common questions.
The source material can be product documentation and previous support tickets.
* **Sales Enablement:** RAG can be used to generate custom responses based on a vast repository of sales documents, case studies, and product specifications.
This allows sales representatives to quickly address potential customer questions and objections
* **Knowledge Management:** RAG enhance the company's internal knowledge base with a bot that can generate answers by pulling information from internal documents, wikis, and databases.
This helps employees find the information they need quickly.
* **Legal Research:** RAG can be used to quickly find relevant legal documents, case law, and statutes to answer legal questions.
* **Medical Diagnosis:** RAG can be used to provide doctors with relevant medical information, research papers, and case studies to help diagnose patients.
* **Code Generation:** RAG can be used to generate code snippets based on a repository of code examples and documentation.
* **Content Creation:** RAG can be used to generate content based on a repository of articles, blog posts, and other written material.
* **Product Development:** Use RAG to retrieve relevant information from past project documentation, market research reports, and customer feedback.
## Use cases where RAG is not the best choice
RAG is a powerful tool, but it is not always the best choice. There are two main cases where RAG is not the best choice:
**When the source material is small and will be used once:** For example, if you need the model to format few paragraphs of text in a table or diagram.
Or you want to turn a short blog into a series of tweets. It will be faster and just as accurate to simply provide the text to the model.
**Summarization:** If you need to summarize a large document, RAG is not the best choice. The retrieval methods will not be able to find relevant chunks with a question
like "summarize this document". And the summary generated by the model will be based on a few random paragraphs from the text. There are summarization models that were specifically trained for this task.
# When to use RAG and when to fine-tune a model
Source: https://thenile.dev/docs/ai-embeddings/rag_vs_finetune
Existing LLM models like GPT-4, Claude, Llama, Gemini and others are trained on a diverse range of data and can generate high-quality responses for a wide range of tasks.
However, they may not be the best choice for tasks that require specialized knowledge, domain-specific information, or a type of problem that requires a different approach.
In such cases, you have two options: use a model that is fine-tuned on a specific dataset or use a model that can retrieve information from a knowledge base.
Comparing RAG (Retrieval Augumented Generation) to fine-tuning is a bit like comparing apples to turkeys. Yes, they are both foods and can be used when you are hungry.
But they are different in many ways - how they are grown, packages, cooked, served, and eaten. And also in their health benefits.
With that in mind, we'll compare RAG and fine-tuning in terms of when to use them, their benefits, and challenges.
While highlighting the fundamental differences between the two approaches.
## When to use RAG
**RAG is a form of prompt engineering.** It is a collection of techniques in which applications retrieve relevant documents and then include them in
the prompt to the generative model.
The goal of RAG is to provide the model with relevant information at the time a question is asked. This allows using LLM with
information that was not available during training (because it is private or new), reminds it of relevant facts and reduces hallucinations.
### Example problems where RAG is a good fit:
* **Customer support:** Automate customer support by providing answers to common questions. The source material can be product documentation and previous support tickets.
* **Sales enablement:** Generate custom responses based on a vast repository of sales documents, case studies, and product specifications.
* **Legal:** Quickly find relevant legal documents, case law, and statutes to answer legal questions.
### Challenges with RAG
RAG's effectiveness depends on the quality of the knowledge base and the retrieval strategy.
In many cases, the knowledge base does not exist and has to be created by painstakingly collecting and structuring data from various sources across a company.
Getting high quality data in a place where it can be easily retrieved isn't a new challenge - it existed, under different names and disciplines
for decades. Many companies have entire data engineering or data platform teams dedicated to this task.
The fact that we have LLMs now helps, but it's not a silver bullet.
The main areas of challenge are:
* Preparing the data for storage and retrieval: Cleaning and structuring the data, chunking large texts,
enriching it with additional information, indexing it for fast retrieval, etc.
* Retrieval strategy: How to retrieve the most relevant documents for a given prompt.
Techniques include semantic similarity using vector embedings, keyword matching, use of relation graphs, and combinations of these.
And of course each technique has variety of algorithms and models with different strengths and different usage patterns.
## When to fine-tune a model
**Fine-tuning a model is a form of training.** It is the process of taking a pre-trained model and training it further on a specific dataset. The goal of the training isn't to get the model to
memorize new data, but rather to learn how to generalize better on a specific task. Because of the generalization process, a fine-tuned model can perform
well on a new task when there isn't a lot of data available or when the relevant data is challenging to retrieve.
### Example problems where fine-tuning is a good fit:
* **Code generation**: If a model wasn't trained on a specific language or framework, it won't perform well in that language, even if you provide examples using RAG.
Code generation doesn't generalize well from a small number of examples, even when they are relevant.
* **Asking customers clarifying questions**. Customers rarely share all available information when they open a support ticket. But most language models are "reluctant" to ask clarifying questions and weren't trained to do so.
RAG isn't useful here because the model isn't missing the data, it is missing a behavior. A large set of example scenarios and which questions are appropriate can train the model with this new behavior.
* **Writing in the style of a specific person**. With enough examples of a specific writing style, a model will generalize and learn to write in that style.
While you could provide examples using RAG, a few examples in the prompt can be costly and hard to generalize from.
* **Generating audio in a specific voice**. An audio generating model can be trained to mimic specific voices.
### Challenges with fine-tuning
Fine-tuning is a form of training, and most of the challenges result from the training process.
* Training requires a large and high quality data set that was prepared specifically for traning the model - separated into "good" and "bad" examples, and covering many variations of the type of tasks the model will perform.
Large number of examples and high variaty is important for generalization (to avoid overfitting). Creating these data sets is time consuming and expensive.
For some use-cases there are public datasets available, which helps. Also worth mentioning that some companies use AI models to generate synthetic data for training.
This is controversial, since it can lead to degredation of the model's performance on real data, however using AI to add variations to real data
can be a good way to create a larger data set and avoid overfitting.
* Training requires a lot of compute power and time. GPUs are expensive and the latest models are hard to acquire at any price.
* Training require not just GPUs, but also use of machine learning tools, languages and libraries. And expertise in using these.
* Training requires expertise beyond just the tools. There are many techniques for training and many hyperparameters to tune. Some techniques are specifically designed for efficiency and can be used even with a single consumer-grade GPU.
These are known as PEFT (Parameter Efficient Model Fine-Tuning), and the most famous of these is called [LORA](https://arxiv.org/pdf/2106.09685).
PEFT only trains a small number of parameters, and if the task requires deeper training, less efficient techniques like full or partial parameter fine-tuning are used.
* Training requires a good process. Knowing how to train a model, how to evaluate it, managing various versions, etc.
* Training can go very wrong very fast. There are a lot of cases where a model performs significantly worse after fine-tuning, and
the training must be restarted from an older version of the model. This behavior is sometimes called "catasrophic forgetting" and is a well-known problem in
machine learning, and this is just one example of how models can quickly and unexpectedly degrade in performance.
* To use a fine-tuned model, you need to deploy it (although some vendors, like OpenAI, allow you to fine-tune on their platform). This is a whole other set of challenges, including how to serve the model, how to monitor it, how to update it.
All this, on top of the GPU costs of serving the model.
## Conclusion
In many cases, choosing between RAG and fine-tuning is trivial. Some problems clearly require just relevant data or clearly require training in new tasks.
However, there are cases where the choice isn't obvious. For example, if you look at online resources for generating SQL from text,
you'll see many who advocate for each one of these approaches.
Generally speaking, RAG is simpler to implement and maintain. Many software engineers already have the skills required or can learn them quickly with some experimentation.
If you get the results you need with RAG, you should use it. If you don't, or if you have the expertise and resources to fine-tune a model, you should consider it.
This investment in fine-tuning could become your competitive edge, as many companies avoid fine-tuning or lack the skills to do it well.
Note that there are also quite a few cases where you could combine both approaches.
For example, in customer support scenario you can use RAG to retrieve relevant documents and use a model that was fine-tuned to ask customers clarifying questions.
Or you can fine-tune a model to generate embeddings for your specific domain, and then use this model in RAG architecture.
In either case, there will be a lot of iterations and experimentation with various models, data sets, and techniques.
# Streaming model responses to the client in real-time
Source: https://thenile.dev/docs/ai-embeddings/streaming
Some generative models offer streaming responses. This greatly improves the user experience by providing ongoing interaction as the model generates the response.
## Getting streaming response from a model
Getting a streaming response from a model is as simple as setting the `stream` parameter to `true` in the model request.
This tells the model to start streaming the response as soon as it starts generating the response.
Here is an example that uses OpenAI:
```javascript theme={null}
import OpenAI from 'openai';
const openai = new OpenAI();
async function main() {
const stream = await openai.chat.completions.create({
model: 'gpt-3.5-turbo',
messages: [{ role: 'user', content: 'Say this is a test' }],
stream: true,
});
for await (const chunk of stream) {
process.stdout.write(chunk.choices[0]?.delta?.content || '');
}
}
main();
```
As you can see, it is straightforward to receive a streaming response from a model. The response is streamed in chunks, and you can process each chunk as it arrives using a simple iterator.
In this case, we printed each chunk to the console.
## Streaming the response from the web framework to the client
Most of the time, however, you would want to stream the response to a client. This means converting the iterator to a ReadableStream (Javascript's Stream API interface), and
then sending the stream to the client via Javascript's Response object.
The conversion can be done with a simple function like this:
```javascript theme={null}
function iteratorToStream(iterator: any) {
return new ReadableStream({
async pull(controller) {
const { value, done } = await iterator.next();
if (done) {
controller.close();
} else {
controller.enqueue(value.content);
}
},
});
}
```
Then you can use this function to send the model response to the client like this:
```javascript theme={null}
const stream = await openai.chat.completions.create({
model: 'gpt-3.5-turbo',
messages: [{ role: 'user', content: 'Say this is a test' }],
stream: true,
});
const stream = iteratorToStream(respStream);
return new Response(stream);
```
## Reading the streaming response in the client
The last step is for the client to display the response as it arrives. This involves:
* Calling the backend API that streams the response
* Reading the stream in chunks
* Calling a hook to update the UI with the new chunk
This is how you can read a streaming response in the client and call a hook to update the UI:
```javascript theme={null}
const resp = await fetch('/api/ask-question', {
body: JSON.stringify({ question: input, tenant_id: tenantid }),
});
// Reader to process the streamed response
if (resp.body) {
const reader = resp.body.getReader();
const decoder = new TextDecoder();
let done = false; // we need to read the stream until it's done
while (!done) {
const { value, done: doneReading } = await reader.read();
done = doneReading;
const chunkValue = decoder.decode(value);
// dispatch function updates the UI with the new chunk
dispatch({ type: 'updateAnswer', text: chunkValue });
if (done) {
dispatch({ type: 'done', text: '' });
}
}
}
```
## Updating the UI with the new chunk
When using React, you can update the UI as each chunk arrives by updating the state with the new chunk. This will cause the component to re-render and display the new chunk.
In this example, we are using a reducer to update the state with the new chunk. The reducer is a function that takes the current state and an action, and returns the new state.
The reducer here has three actions: `addQuestion`, `updateAnswer`, and `done`. The `addQuestion` action adds a new question to the conversation,
the `updateAnswer` action updates the answer with the new chunk, and the `done` action marks the end of the last answer.
Note that we are creating a new array with the new chunk and updating the state with this new array, if you reuse the existing array, React will not re-render the component.
```javascript theme={null}
type AppActions = {
type: string,
text: string,
};
interface AppState {
messages: MessageType[] | [];
}
function reducer(state: AppState, action: AppActions): AppState {
switch (action.type) {
case "addQuestion":
return {
...state,
messages: [
...state.messages,
{ type: "question", text: action.text },
{ type: "answer", text: "" },
],
};
case "updateAnswer":
const conversationListCopy = [...state.messages];
const lastIndex = conversationListCopy.length - 1;
conversationListCopy[lastIndex] = {
...conversationListCopy[lastIndex],
text: conversationListCopy[lastIndex].text + action.text,
};
return {
...state,
messages: conversationListCopy,
};
case "done":
return {
...state,
};
default:
return state;
}
}
```
Now we need to tie the reducer to the component. We can do this using the `useReducer` hook. The `useReducer` hook takes the reducer function and the initial state,
and returns the current state and a dispatch function. The hook has two parts - the state and the dispatch function. The state is the current state of the component,
and we use this to show the current conversation. The dispatch function is used to send actions to the reducer, which updates the state.
As you saw in an earlier snippet, we call `dispatch` with the action when we read a new chunk from the stream.
The snippet below shows how to use the `useReducer` hook to update the state with the new chunk, and how to use the state in the component to display the conversation.
```javascript theme={null}
const Chatbox: React.FC = () => {
const [state, dispatch] = useReducer(reducer, { messages: [] });
// ... lots of UI code here....
{state.messages.map((msg, index) => (
{
// display the message, this will get updated as the response streams
msg.text
}
))}
;
};
```
## Structured streaming responses
If you are used to writing web applications, you are probably used to sending structured responses like JSON.
JSON allows you to package multiple pieces of information in a single response, and easily handle all the information in the client.
Therefore, it is important to note that JSON and streaming responses don't mix well. The problem is that JSON structure is unparsable until the client recieves the very last `}` of the response.
Which means that the client will not be able to parse the response until the very end, and will not be able to display the response until the very end.
Which is the opposite of what we want with streaming responses.
To solve this, you need to use a slightly different format. A common way to handle this is to have structured metadata at the beginning of the response, and then stream the content.
You also need a unique delimiter to separate the metadata from the content. This way, the client can parse the metadata and start displaying the content as it arrives.
Here is an example of how you can structure the response that you send from the backend.
First, the `iteratorToStream` function needs to be updated to send the metadata. We'll modify it to accept a metadata string (we'll `stringify` the JSON to generate it),
and send the metadata and the delimiter when the stream starts, before any of the iterator chunks are streamed:
```javascript theme={null}
function iteratorToStream(iterator: any, metadata: string) {
return new ReadableStream({
start(controller) {
controller.enqueue(metadata);
// this is our separator. Streaming answer comes next
controller.enqueue("EOJSON");
},
async pull(controller) {
const { value, done } = await iterator.next();
if (done) {
controller.close();
} else {
controller.enqueue(value.content);
}
},
});
}
```
Then, we need to update the way the client parses the response. We need to wait for all the metadata to arrive, parse it, and then start displaying the content:
```javascript theme={null}
if (resp.body) {
const reader = resp.body.getReader();
const decoder = new TextDecoder();
let done = false;
// accumulate the data as it comes in, so we can parse the metadata
let partialData = '';
let recievedMetadata = false; // track if we finished with the JSON yet
while (!done) {
const { value, done: doneReading } = await reader.read();
done = doneReading;
const chunkValue = decoder.decode(value);
if (!recievedMetadata) {
partialData += chunkValue;
// first part of the response is json, second part is the answer
const dataParts = partialData.split('EOJSON');
if (dataParts.length > 1) {
const metadata = JSON.parse(dataParts[0]);
// use the metadata, for example to update the UI with the model name
handleMetadata(metadata);
// handle the first chunk of the answer
dispatch({ type: 'updateAnswer', text: dataParts[1] });
recievedFiles = true;
partialData = '';
}
} else {
// handle the rest of the answer
dispatch({ type: 'updateAnswer', text: chunkValue });
if (done) {
dispatch({ type: 'done', text: '' });
}
}
}
}
```
# DiskANN
Source: https://thenile.dev/docs/ai-embeddings/vectors/diskann
DiskANN index support for PostgreSQL with pgvectorscale
The `pgvectorscale` extension adds diskANN index support for pgvector.
This extension is useful in cases where `pgvector`'s `hnsw` index does not fit into available memory and as a result the ANN search does not perform as expected.
## Key Features
* StreamingDiskANN index - disk-backed HNSW variant.
* Statistical Binary Quantization (SBQ)
* Label-based filtering combined with DiskANN index.
## Example: DiskANN index on shared table
To keep the example readable we'll work with **3-dimensional vectors**.
Swap `VECTOR(3)` for `VECTOR(768)` or `VECTOR(1536)` in real apps.
```sql theme={null}
-- 1. Shared data table
CREATE TABLE document_embedding (
id BIGSERIAL PRIMARY KEY,
contents TEXT,
metadata JSONB,
embedding VECTOR(3)
);
-- 2. Seed with tiny sample data
INSERT INTO document_embedding (contents, metadata, embedding) VALUES
('T-shirt', '{"category":"apparel"}', '[0.10, 0.20, 0.30]'),
('Sweater', '{"category":"apparel"}', '[0.12, 0.18, 0.33]'),
('Coffee mug', '{"category":"kitchen"}', '[0.90, 0.80, 0.70]');
-- 3. Build a DiskANN index (cosine distance)
CREATE INDEX document_embedding_diskann_idx
ON document_embedding
USING diskann (embedding vector_cosine_ops);
-- 4. k-NN query (top-2 similar items)
SELECT id, contents, metadata
FROM document_embedding
ORDER BY embedding <=> '[0.11, 0.21, 0.29]' -- query vector
LIMIT 2;
```
You should see the two apparel rows first - a good sanity check that the index works.
## Example: DiskANN index on tenant-aware table
```sql theme={null}
-- 1. Tenant-aware table
CREATE TABLE tenant_embedding (
tenant_id UUID NOT NULL,
doc_id BIGINT,
embedding VECTOR(2), -- using tiny 2‑dim vectors for demo
metadata JSONB,
PRIMARY KEY (tenant_id, doc_id)
);
-- 2. Create some tenants
INSERT INTO tenants (id, name) VALUES
('11111111-1111-1111-1111-111111111111', 'Tenant A');
INSERT INTO tenants (id, name) VALUES
('22222222-2222-2222-2222-222222222222', 'Tenant B');
-- 3. Seed soome data
INSERT INTO tenant_embedding (tenant_id, doc_id, embedding, metadata) VALUES
('11111111-1111-1111-1111-111111111111', 1, '[0.05, 0.95]', '{"title":"DocA"}'),
('11111111-1111-1111-1111-111111111111', 2, '[0.04, 0.90]', '{"title":"DocB"}');
INSERT INTO tenant_embedding (tenant_id, doc_id, embedding, metadata) VALUES
('22222222-2222-2222-2222-222222222222', 1, '[0.80, 0.20]', '{"title":"DocC"}');
-- 3. Create an index (Nile will partition by tenant_id)
CREATE INDEX tenant_embedding_diskann_idx
ON tenant_embedding
USING diskann (embedding vector_cosine_ops);
-- 4. Tenant‑scoped ANN query
SET nile.tenant_id = '11111111-1111-1111-1111-111111111111';
SELECT doc_id, metadata
FROM tenant_embedding
ORDER BY embedding <=> '[0.06, 0.92]'
LIMIT 2;
```
## Example: Label-based filtering
Label-based filtering is a technique that allows you to filter the results of an ANN search based on a label while using the DiskANN index.
Other filters are supported, but will use pgvector's post-filtering (i.e. after the ANN search).
In order to use label based filtering, you need to:
* Create a label column in your table. It has to be an array of `smallint`s. Other types will revert to using the post-filtering.
* Create a diskann index that uses both the embedding and the label column.
* Use the `&&` (array intersection) operator in search queries.
* Optional, but recommended: Use a separate table and joins to translate smallint labels to meaningful descriptions.
```sql theme={null}
-- 1. Create a label column
CREATE TABLE documents (
id BIGSERIAL PRIMARY KEY,
embedding VECTOR(3),
labels SMALLINT[]
);
-- 2. Create an index on the label column
-- Insert a couple of demo rows
INSERT INTO documents (embedding, labels) VALUES
('[0.3,0.2,0.1]', ARRAY[1]), -- label 1 = science
('[0.35,0.25,0.05]', ARRAY[1,2]), -- label 2 = business
('[0.9,0.8,0.7]', ARRAY[3]); -- label 3 = art
-- 3. Create an index on the label column
CREATE INDEX documents_ann_idx
ON documents
USING diskann (embedding vector_cosine_ops, labels);
-- 4. Query with label-based filtering
SELECT *
FROM documents
WHERE labels && ARRAY[1,2]
ORDER BY embedding <=> '[0.32,0.18,0.12]'
LIMIT 5;
-- 5. Optional: Translate labels to descriptions
CREATE TABLE labels (
id SMALLINT PRIMARY KEY,
description TEXT
);
INSERT INTO labels (id, description) VALUES
(1, 'Science'),
(2, 'Business'),
(3, 'Art');
-- 6. Query with label-based filtering and description
SELECT d.*
FROM documents d
WHERE d.labels && (
SELECT array_agg(id)
FROM labels
WHERE description in ('Science', 'Business')
)
ORDER BY d.embedding <=> '[0.32,0.18,0.12]'
LIMIT 5;
```
## Limitations
* DiskANN index supports `cosine`, `l2` and `inner_product` distance metrics, not the entire pgvector's set of distance metrics.
* Label-based filtering is only supported for `smallint` arrays and the `&&` operator. Other types will revert to using the post-filtering.
* DiskANN is best suited for datasets where `hnsw` index would be too large to fit into memory. For smaller datasets, `hnsw` is still a good choice.
## Additional Resources
[Pgvectorscale github repository](https://github.com/timescale/pgvectorscale)
# Getting started with pgvector
Source: https://thenile.dev/docs/ai-embeddings/vectors/pg_vector
The **`pgvector`** extension in PostgreSQL is used to efficiently store and query vector data. The **`pgvector`** extension provides
PostgreSQL with the ability to store and perform operations on vectors directly within the database.
Nile supports **`pgvector`** out of the box on the latest version - `0.8.0`.
Pgvector lets you store and query vectors directly within your usual Postgres database - with the rest of your data. This is both convenient and efficient. It supports:
* Exact and approximate nearest neighbor search (with optional HNSW and IVFFlat indexes)
* Single-precision, half-precision, binary, and sparse vectors
* L2 distance, inner product, cosine distance, L1 distance, Hamming distance, and Jaccard distance
* Any language with a Postgres client
Plus ACID compliance, point-in-time recovery, JOINs, and all of the other great features of Postgres
## Create tenant table with vector type
Vector types work like any other standard types. You can make them the type of a column in a tenant table and Nile will take care of isolating the embeddings per tenant.
```sql theme={null}
-- creating a table to store wiki documents for a Notion like
-- SaaS application with vector dimension of 3
CREATE TABLE wiki_documents(
tenant_id uuid,
id integer,
embedding vector(3)
);
```
## Store vectors per tenant
Once you have the table defined, you would want to populate the embeddings. Typically, this is done by querying a large language model (eg. OpenAI, HuggingFace), retrieving the embeddings and storing them in the vector store. Once stored, the embeddings follow the standard tenant rules. They can be isolated, sharded and placed based on the tenant they belong to.
```sql theme={null}
INSERT INTO wiki_documents (tenant_id,id, embedding)
VALUES ('018ade1a-7843-7e60-9686-714bab650998',1, '[1,2,3]');
```
## Query vectors
Pgvector supports 6 types of vector similarity operators:
Operator
Name
Description
Use Cases
\<->
vector\_l2\_ops
L2 distance. Measure of the straight-line distance between two points in
a multi-dimensional space. It calculates the length of the shortest path
between the points, which corresponds to the hypotenuse of a right
triangle.
Used in clustering, k-means clustering, and distance-based
classification algorithms
\<#>
vector\_ip\_ops
Inner product. The inner product, also known as the dot product,
measures the similarity or alignment between two vectors. It calculates
the sum of the products of corresponding elements in the vectors.
Used in similarity comparison or feature selection. Note that for
normalized vectors, inner product will result in the same ranking as
cosine distance, but is more efficient to calculate. So this is a good
choice if you use an embedding algorith that produces normalized vectors
(such as OpenAI's)
\<=>
vector\_cosine\_ops
Cosine distance. Cosine distance, often used as cosine similarity when
measuring similarity, quantifies the cosine of the angle between two
vectors in a multi-dimensional space. It focuses on the direction rather
than the magnitude of the vectors.
Used in text similarity, recommendation systems, and any context where
you want to compare the direction of vectors
\<+>
vector\_l1\_ops
L1 distance. The L1 distance, also known as the Manhattan distance,
measures the distance between two points in a grid-like path (like a
city block). It is the distance between two points measured along axes
at right angles.
Less sensitive to outliers than L2 distance and according to some
research, better for high-dimensional data.
\<\~>
bit\_hamming\_ops
Hamming distance. The Hamming distance measures the number of positions
at which the corresponding symbols are different.
Used with binary vectors. Mostly for discrete data like categories. Also
used for error-correcting codes and data compression.
\<%>
bit\_jaccard\_ops
Jaccard distance. Measures similarity between sets by calculating the
ratio of the intersection to the union of the two sets (how many
positions are the same out of the total positions).
Used with binary vectors. Useful for comparing customers purchase
history, recommendation systems, similarites in terms used in different
texts, etc.
You could use any one of them to find the distance between vectors. The choice of operator depends not only on the use-case, but also on
the model used to generate the embeddings. For example, OpenAI's models return normalized embeddings, so using inner product or cosine distance
will give the same results and inner product is more efficient. However, some models return non-normalized embeddings, so cosine distance should be used.
Real world vectors are quite large - 768 to 4096 dimensions are not uncommon. But for the sake of the example, we'll use small vectors:
To get the L2 distance
```sql theme={null}
SELECT embedding <-> '[3,1,2]' AS distance FROM wiki_documents;
```
For inner product, multiply by -1 (since `<#>` returns the negative inner product)
```sql theme={null}
SELECT (embedding <#> '[3,1,2]') * -1 AS inner_product FROM wiki_documents;
```
For cosine similarity, use 1 - cosine distance
```sql theme={null}
SELECT 1 - (embedding <=> '[3,1,2]') AS cosine_similarity FROM wiki_documents;
```
## Vector Indexes
`pgvector` supports two types of indexes:
* HNSW
* IVFFlat
Keep in mind that vector indexes are unlike other database indexes. They are used to perform efficient **approximate** nearest neighbor searches.
Without vector indexes (also called **flat indexes** by other vector stores), queries sequentially scan through all vectors for the given query,
and compute the distance to the query vector. This is computationally expensive, but guarantees to find the nearest neighbors
(also called **exact nearest neighbors** or **KNN**).
With vector indexes, the query will search a subset of the vectors that is expected to contain the nearest neighbors (but may not contain all of them).
This is computationally efficient, but the results are not guaranteed to be the nearest neighbors.
When using vector indexes, you can control the trade-off between speed and recall by specifying the index type and parameters.
### HNSW
An HNSW index creates a multilayer graph. It has slower build times and uses more memory than IVFFlat, but has better query performance
(in terms of speed-recall tradeoff). There’s no training step like IVFFlat, so the index can be created without any data in the table.
When creating HNSW, you can specify the maximum number of connections in a layer (`m`) and the number of candidate vectors considered
when building the graph (`ef_construction`). More connections and more candidate vectors will improve recall but will increase build time and memory.
If you don't specify the parameters, the default values are `m = 16` and `ef_construction = 64`.
Add an index for each distance function you want to use.
```sql theme={null}
CREATE INDEX ON wiki_documents USING hnsw (embedding vector_l2_ops);
```
Inner product
```sql theme={null}
CREATE INDEX ON wiki_documents USING hnsw (embedding vector_ip_ops);
```
Cosine distance
```sql theme={null}
CREATE INDEX ON wiki_documents USING hnsw (embedding vector_cosine_ops);
```
Specifying the parameters
```sql theme={null}
CREATE INDEX ON wiki_documents USING hnsw (embedding vector_cosine_ops) WITH (m = 16, ef_construction = 100);
```
While querying, you can specify the size of the candidate list that will be searched (`hnsw_ef`):
```sql theme={null}
SET hnsw_ef = 100;
SELECT * FROM wiki_documents ORDER BY embedding <=> '[3,1,2]' LIMIT 10;
```
Vectors with up to 2,000 dimensions can be indexed.
### IVFLAT
An IVFFlat index divides vectors into lists, and then searches a subset of those lists that are closest to the query vector.
It has faster build times and uses less memory than HNSW, but has lower query performance (in terms of speed-recall tradeoff).
Three keys to achieving good recall are:
1. Create the index **after** the table has some data
2. Choose an appropriate number of lists - a good place to start is `rows / 1000` for up to 1M rows and `sqrt(rows)` for over 1M rows.
3. When querying, specify an appropriate number of probes (higher is better for recall, lower is better for speed) - a good place to start is `sqrt(lists)`
Add an index for each distance function you want to use.
L2 distance
```sql theme={null}
CREATE INDEX ON wiki_documents USING ivfflat (embedding vector_l2_ops) WITH (lists = 100);
```
Inner product
```sql theme={null}
CREATE INDEX ON wiki_documents USING ivfflat (embedding vector_ip_ops) WITH (lists = 100);
```
Cosine distance
```sql theme={null}
CREATE INDEX ON wiki_documents USING ivfflat (embedding vector_cosine_ops) WITH (lists = 100);
```
When querying, you can specify the number of probes (i.e. how many lists to search):
```sql theme={null}
SET ivfflat_probes = 100;
SELECT * FROM wiki_documents ORDER BY embedding <=> '[3,1,2]' LIMIT 10;
```
More probes will improve recall but will slow down the query.
Vectors with up to 2,000 dimensions can be indexed.
## Filtering
Typically, vector search is used to find the nearest neighbors, which means that you would limit the number after ordering by distance:
```sql theme={null}
SELECT * FROM wiki_documents ORDER BY embedding <=> '[3,1,2]' LIMIT 10;
```
It is a good idea to also filter by distance, so you won't include vectors that are too far away, even if they are the nearest neighbors.
This will prevent you from getting nonsensical results in cases where there isn't much data in the vector store.
```sql theme={null}
SELECT * FROM wiki_documents
WHERE embedding <=> '[3,1,2]' < 0.9
ORDER BY embedding <=> '[3,1,2]'
LIMIT 10;
```
Nile will automatically limit the results to only the vectors that belong to the current tenant:
```sql theme={null}
SET nile.tenant_id = '018ade1a-7843-7e60-9686-714bab650998';
SELECT * FROM wiki_documents
WHERE embedding <=> '[3,1,2]' < 0.9
ORDER BY embedding <=> '[3,1,2]'
LIMIT 10;
```
And you can also filter results by additional criteria (some vector stores call this "metadata filtering" or "post-filtering"):
```sql theme={null}
SELECT * FROM wiki_documents
WHERE embedding <=> '[3,1,2]' < 0.9
AND category = 'product'
ORDER BY embedding <=> '[3,1,2]'
LIMIT 10;
```
It is recommended to add indexes on the columns used for filtering, so the query can be optimized.
Especially when the use of the filter can lead to small enough result sets that a sequential scan is faster than use of the vector index.
This will optimize not just performance but also recall.
With approximate indexes, filtering is applied after the index is scanned. If a condition matches 10% of rows,
with HNSW and the default hnsw\.ef\_search of 40, only 4 rows will match on average.
Starting in version `0.8.0`, you can enable **iterative index scans**, which will automatically scan more of the index when needed.
### Iterative Index scans
Using iterative index scans, Postgres will scan the approximate index for nearest neighbors, apply additional filters and, if the number of
neighbors after filtering is insufficient, it will continue scanning until sufficient results are found.
Each index has its own configuration (GUC) for iterative scans: `hnsw.iterative_scan` and `ivfflat.iterative_scan`.
By default both configurations are set to `off`.
HNSW indexes support both relaxed and strict ordering for the iterative scans. Strict order guarantees that the returned results are ordered by exact distance.
Relaxed order allows results that are slightly out of order, but provides better recall (i.e. fewer missed results due to the approximate
nature of the index).
```sql theme={null}
SET hnsw.iterative_scan = strict_order;
-- or
SET hnsw.iterative_scan = relaxed_order;
```
IVFFlat indexes only allow relaxed ordering:
```sql theme={null}
SET ivfflat.iterative_scan = relaxed_order;
```
Once you set these configs, you don't need to change your existing queries. You should immediately see the same queries return the correct number of results.
However, if you use relaxed ordering, you can re-order the result using materialized CTE:
```sql theme={null}
WITH relaxed_results AS MATERIALIZED (
SELECT id, embedding <-> '[1,2,3]' AS distance FROM items WHERE category_id = 123 ORDER BY distance LIMIT 5
) SELECT * FROM relaxed_results ORDER BY distance;
```
If you filter by distance (recommended, to avoid nonsense results in case there aren't many similar vectors), it is recommended to use
materialized CTE and place the filter outside the CTE in order to avoid overscanning:
```sql theme={null}
WITH nearest_results AS MATERIALIZED (
SELECT id, embedding <=> '[1,2,3]' AS distance FROM items ORDER BY distance LIMIT 5
) SELECT * FROM nearest_results WHERE distance < 1 ORDER BY distance;
```
Even with iterative scans, pgvector limits the index scan in order to balance time, resource use and recall.
Increasing these limits can increase query latency but potentially improve recall:
HSNW has a configuration that controls the total number of rows that will be scanned by a query across all iterations (20,000 is the default):
```sql theme={null}
SET hnsw.max_scan_tuples = 20000;
```
IVFFLat lets you configure the maximum number of lists that will be checked for nearest neighbors:
```sql theme={null}
SET ivfflat.max_probes = 100;
```
## Quantization
*Introduced in pgvector 0.7.0*
Quantization is a technique of optimizing vector storage and query performance by using fewer bits to store the vectors.
By default, pgvector's Vector type is in 32-bit floating point format. The `halfvec` data type uses 16-bit floating point format, which has the following benefits:
* Reduced storage requirements (half the memory)
* Faster query performance
* Reduced index size (both in disk and memory)
* Can index vectors with up to 4096 dimensions (which covers the most popular embedding models)
To use `halfvec`, you can create a table with the `halfvec` type:
```sql theme={null}
CREATE TABLE wiki_documents(
tenant_id uuid,
id integer,
embedding halfvec(3) -- put the real number of dimensions here
);
```
You can also create quantized indexes on regular sized vectors:
```sql theme={null}
CREATE INDEX ON wiki_documents USING hnsw ((embedding::halfvec(3)) halfvec_l2_ops);
```
In this case, you need to cast the vector to `halfvec` in the query:
```sql theme={null}
SELECT * FROM wiki_documents WHERE embedding::halfvec(3) <=> '[3,1,2]' LIMIT 10;
```
Note that to create an index on `halfvec`, you need to specify the distance function as `halfvec_l2_ops` or `halfvec_cosine_ops`.
## Sparse Vectors
*Introduced in pgvector 0.7.0*
Sparse vectors are vectors in which the values are mostly zero. These are common in text search algorithms, where each dimension represents
a word and the value represents the relative frequency of the word in the document - BM25, for example. Some embedding models, such as BGE-M3, also use sparse vectors.
Pgvector supports sparse vector type `sparsevec` and the associated similarity operators.
Because sparse vectors can be extremely large but most of the values are zero, pgvector stores them in a compressed format.
```sql theme={null}
CREATE TABLE wiki_documents(
tenant_id uuid,
id integer,
embedding sparsevec(5) -- put the real number of dimensions here
);
INSERT INTO wiki_documents (tenant_id, id, embedding)
VALUES ('018ade1a-7843-7e60-9686-714bab650998', 1, '{1:1,3:2,5:3}/5');
SELECT * FROM wiki_documents ORDER BY embedding <-> '{1:3,3:1,5:2}/5' LIMIT 5;
```
The format is `{index1:value1,index2:value2,...}/N`, where N is the number of dimensions and the indices start from 1 (like SQL arrays).
Because the format is a bit unusual, it is recommended to use [pgvector's libraries for your favorite language](https://github.com/pgvector/pgvector?tab=readme-ov-file#languages)
to insert and query sparse vectors.
## Summary
You can read more about pgvector on their [github](https://github.com/pgvector/pgvector/blob/master/README.md)
If you have any feedback or questions on building AI-native SaaS applications on Nile, please do reach out on our [Github discussion forum](https://github.com/orgs/niledatabase/discussions) or our [Discord community](https://discord.gg/8UuBB84tTy).
# Vector Database
Source: https://thenile.dev/docs/ai-embeddings/vectors/vector_database
### **What is a Vector Database?**
A **vector database** is a specialized type of database designed to store and search **vectors**, which are numerical representations of data. These vectors capture the essential characteristics of complex data like text, images, or audio. For example, in a **natural language processing (NLP)** task, a sentence like "The quick brown fox jumps over the lazy dog" can be converted into a vector, such as `[0.12, -0.34, 0.67, -0.23, ...]`, where each number represents specific features of the sentence, like its semantic meaning. Similarly, in an **image recognition** task, an image of a dog can be represented as a vector, like `[0.45, 0.78, -0.32, 0.67, ...]`, where each value encodes key attributes of the image such as texture, color, or shape.
Vector databases are optimized for **similarity searches**, meaning they help you find data points that are "close" to each other in terms of meaning or features, rather than exact matches. For instance, if you're running a **search engine** for articles, and you input the phrase "machine learning applications," the query is converted into a vector. The database then searches for articles whose vectors are similar to the query vector, retrieving content like "AI use cases in healthcare" or "Deep learning in robotics," even if the exact phrase “machine learning applications” isn’t present. Similarly, in an **e-commerce** recommendation system, when a user views a product like a red T-shirt, its vector representation is used to find and recommend similar products, such as other T-shirts with similar color or style.
***
### **Core Concepts and Examples**
1. **Vectors**: A vector is a list of numbers that represents the essential characteristics of data. In AI, vectors are created using models that capture relationships between elements of the data.
* **Example in NLP**: A sentence like "The cat sat on the mat" can be converted into a vector by a language model such as BERT or GPT. The vector might look something like `[0.34, 0.67, -0.23, 0.88, ...]`. This vector contains semantic information about the sentence.
* **Example in Image Recognition**: An image of a car could be transformed into a vector that represents its visual features, such as color, shape, and texture: `[0.23, -0.12, 0.98, 0.34, ...]`.
2. **Similarity Search**: The main task of a vector database is to find vectors that are "close" to each other in a high-dimensional space. For example, you may want to find images that are visually similar to a query image or articles that are semantically similar to a query sentence.
* **Example in Product Recommendations**: If a user looks at a product on an e-commerce website, the system converts the product into a vector. The vector database is then queried to find vectors of similar products, which are displayed as recommendations.
* **Example in Large Language Models (LLMs)**: After processing a user query like "What is quantum computing?", the LLM generates a vector representing the query’s meaning. The vector database is queried to find documents or articles with similar meaning, even if the exact words differ.
3. **Nearest Neighbor Search**: Vectors in a vector database are stored in such a way that when you search for one vector, the database can efficiently find the "nearest" vectors, based on a distance metric such as **cosine similarity** or **Euclidean distance**.
* **Example in Music Recommendation**: A music streaming app converts songs into vectors based on their audio features (e.g., tempo, rhythm). When a user likes a song, the app searches the vector database to find similar songs.
4. **Indexing in Vector Databases**: To search efficiently in high-dimensional spaces, vector databases use special indexing methods such as **Hierarchical Navigable Small World (HNSW)** or **IVF (Inverted File Index)**.
***
### **How Vector Databases Work with Large Language Models (LLMs)**
Large Language Models, such as GPT-4 or BERT, generate vectors (embeddings) from text that represent the semantic meaning of the input. These embeddings can be stored in vector databases for efficient retrieval in tasks like semantic search, question-answering, and more.
### **Example 1: Semantic Search with LLMs**
When a user enters a query like "Best way to learn machine learning," an LLM transforms this query into a vector embedding that captures the essence of the question. This vector is then passed to the vector database, which searches for similar vectors (e.g., documents, articles, blog posts) related to machine learning education.
* **How it works**:
1. LLM converts the query into a vector.
2. The vector database performs a similarity search to find the closest matching vectors.
3. The system retrieves and returns relevant documents, even if they don’t contain the exact words “best way to learn machine learning” but discuss similar concepts.
### **Example 2: Document Search and Retrieval**
Imagine a legal firm storing thousands of legal documents. Each document is processed by a transformer model, generating a vector that represents its content. These vectors are then stored in a vector database. When a user searches for "case law on intellectual property infringement," the query is transformed into a vector, and the database retrieves documents that are semantically similar.
* **How it works**:
1. The LLM transforms each document into a vector embedding at the time of ingestion.
2. When the user performs a search, the query is also converted into a vector.
3. The vector database searches for documents with vectors close to the query vector, effectively finding relevant documents.
### **Example 3: Chatbots with LLMs and Vector Databases**
In an LLM-based chatbot, user queries can be mapped into vector embeddings, and these embeddings can be stored for future use. If the user asks a similar question later, the chatbot can use the vector database to retrieve past responses based on vector similarity, enabling more coherent and contextually aware responses.
* **How it works**:
1. User query gets converted into a vector by the LLM.
2. The chatbot stores this vector in a vector database alongside the generated response.
3. Future similar queries are compared with vectors in the database to retrieve the most relevant responses.
***
### **Use cases for Vector Databases**
1. **Recommendation Engines**:
* **Use Case**: A movie streaming platform uses a vector database to store user preferences and movie vectors (e.g., genre, ratings). When a user watches or likes a movie, the system searches for movies with similar vectors.
* **Example**: The user likes a drama film with vectors such as `[0.45, -0.32, 0.78, 0.11]`. The vector database returns films with vectors close to this one.
2. **Image Search**:
* **Use Case**: A social media platform uses vector databases to store image embeddings. When users upload an image, the platform uses a deep learning model to generate a vector for the image. A similarity search is then performed to find images with similar vectors.
* **Example**: A user uploads a picture of a sunset, and the vector database retrieves other sunset images, even though they are different in pixel composition.
3. **Fraud Detection**:
* **Use Case**: In financial services, transaction histories are represented as vectors. Fraud detection systems use vector databases to find anomalous behavior by searching for transactions whose vectors deviate significantly from normal patterns.
* **Example**: A fraudulent transaction might produce a vector that is distant from typical transaction vectors. The system flags it as suspicious based on vector distance.
4. **Search in Chatbots**:
* **Use Case**: LLMs combined with vector databases are used in chatbots to improve user query resolution. When a user asks a question, the chatbot converts the query into a vector and searches a vector database to retrieve the most relevant answer.
* **Example**: A user asks, "What are the top programming languages?" The LLM produces a vector for the query and retrieves responses that contain information about programming languages like Python, Java, and C++.
***
### **Challenges of Vector Databases**
1. **Curse of Dimensionality**: As the number of dimensions in vector data increases, it becomes harder to distinguish between similar and dissimilar vectors. This can make searches less effective unless carefully managed.
2. **Approximation vs. Accuracy**: Vector databases often use **approximate nearest-neighbor (ANN)** search techniques to improve performance. This can sacrifice accuracy, especially in highly sensitive applications, in favor of speed.
3. **Memory and Storage**: High-dimensional vector data can take up significant amounts of memory and storage. Managing this efficiently is crucial for scaling vector databases to handle millions or billions of vectors.
# Create a database
Source: https://thenile.dev/docs/api-reference/databases/create-a-database
https://global.thenile.dev/openapi.yaml post /workspaces/{workspaceSlug}/databases
Creates a database record in the control plane and triggers creation in the
target region. Database names must be less than 64 characters, and unique
within a workspace. Names may contain letters (lower or uppercase), numbers,
and underscores, and must begin with a letter or underscore.
# Create a database credential
Source: https://thenile.dev/docs/api-reference/databases/create-a-database-credential
https://global.thenile.dev/openapi.yaml post /workspaces/{workspaceSlug}/databases/{databaseName}/credentials
Generates a new secure credential for a database. The generated password is
returned in the response, and is the only time the password is available in
plaintext. No request body is required."
# Create a dedicated compute instance
Source: https://thenile.dev/docs/api-reference/databases/create-a-dedicated-compute-instance
https://global.thenile.dev/openapi.yaml post /workspaces/{workspaceSlug}/databases/{databaseName}/compute
# Delete a database
Source: https://thenile.dev/docs/api-reference/databases/delete-a-database
https://global.thenile.dev/openapi.yaml delete /workspaces/{workspaceSlug}/databases/{databaseName}
Marks a database as deleted and queues its data for deletion. The database will
no longer appear in your Nile dashboard. You will be able to connect to it and
operate on it while it is queued for deletion, but compute costs still apply.
# Delete a database credential
Source: https://thenile.dev/docs/api-reference/databases/delete-a-database-credential
https://global.thenile.dev/openapi.yaml delete /workspaces/{workspaceSlug}/databases/{databaseName}/credentials/{credentialId}
Deletes an existing database credential. No request body is required."
# Delete a dedicated compute instance
Source: https://thenile.dev/docs/api-reference/databases/delete-a-dedicated-compute-instance
https://global.thenile.dev/openapi.yaml delete /workspaces/{workspaceSlug}/databases/{databaseName}/compute/{instanceId}
Removes a compute instance from your database and queues it for deletion. The
instance will no longer appear in your Nile dashboard. When deletion is
finalized, all data associated with the instance will be deleted.
# Describe a dedicated compute instance
Source: https://thenile.dev/docs/api-reference/databases/describe-a-dedicated-compute-instance
https://global.thenile.dev/openapi.yaml get /workspaces/{workspaceSlug}/databases/{databaseName}/compute/{instanceId}
# Get a database
Source: https://thenile.dev/docs/api-reference/databases/get-a-database
https://global.thenile.dev/openapi.yaml get /workspaces/{workspaceSlug}/databases/{databaseName}
Gets details of a database.
# List available regions
Source: https://thenile.dev/docs/api-reference/databases/list-available-regions
https://global.thenile.dev/openapi.yaml get /workspaces/{workspaceSlug}/regions
Retrieve the list of available region identifiers for a workspace.
Identifiers are composed of a prefix indicating the underlying cloud provider
followed by a region name in that provider. For example `AWS_US_WEST_2` is
associated with the `us-west-2` region (Portland, OR) of AWS.
# List databases
Source: https://thenile.dev/docs/api-reference/databases/list-databases
https://global.thenile.dev/openapi.yaml get /workspaces/{workspaceSlug}/databases
Lists all of the databases in a workspace.
# List dedicated compute instances for a database
Source: https://thenile.dev/docs/api-reference/databases/list-dedicated-compute-instances-for-a-database
https://global.thenile.dev/openapi.yaml get /workspaces/{workspaceSlug}/databases/{databaseName}/compute
# Lists credentials for a database
Source: https://thenile.dev/docs/api-reference/databases/lists-credentials-for-a-database
https://global.thenile.dev/openapi.yaml get /workspaces/{workspaceSlug}/databases/{databaseName}/credentials
Lists all credentials that exist for a database. The id of the credential is
included in the response, but not the password. Passwords are provided only once,
at the time the credential is created, and are encrypted for storage.
# Rotate a database credential
Source: https://thenile.dev/docs/api-reference/databases/rotate-a-database-credential
https://global.thenile.dev/openapi.yaml post /workspaces/{workspaceSlug}/databases/{databaseName}/credentials/rotate
Generates a new credential and schedules existing credentials for deletion after the provided delay.
# Update a dedicated compute instance
Source: https://thenile.dev/docs/api-reference/databases/update-a-dedicated-compute-instance
https://global.thenile.dev/openapi.yaml put /workspaces/{workspaceSlug}/databases/{databaseName}/compute/{instanceId}
Rename and/or resize a dedicated compute instance.
# Deletes a previously issued invite
Source: https://thenile.dev/docs/api-reference/developers/deletes-a-previously-issued-invite
https://global.thenile.dev/openapi.yaml delete /workspaces/{workspaceSlug}/invites/{inviteId}
# Identify developer details
Source: https://thenile.dev/docs/api-reference/developers/identify-developer-details
https://global.thenile.dev/openapi.yaml get /developers/me
Returns information about the developer associated with the token provided,
including the workspaces and database ids the developer is authorized to access,
and the email address associated with the developer.
# Invite a developer to a workspace
Source: https://thenile.dev/docs/api-reference/developers/invite-a-developer-to-a-workspace
https://global.thenile.dev/openapi.yaml post /workspaces/{workspaceSlug}/invites
Sends an email to another developer with a link that allows them to join the
workspace.
# List developer invites
Source: https://thenile.dev/docs/api-reference/developers/list-developer-invites
https://global.thenile.dev/openapi.yaml get /workspaces/{workspaceSlug}/invites
Lists all developer invites in a workspace
# Post oauth2token
Source: https://thenile.dev/docs/api-reference/post-oauth2token
https://global.thenile.dev/openapi.yaml post /oauth2/token
# Change level (effective now or at timestamp)
Source: https://thenile.dev/docs/api-reference/workspaces/change-level-effective-now-or-at-timestamp
https://global.thenile.dev/openapi.yaml put /workspaces/{workspaceSlug}/subscription
# Close a subscription row at given time (now if omitted)
Source: https://thenile.dev/docs/api-reference/workspaces/close-a-subscription-row-at-given-time-now-if-omitted
https://global.thenile.dev/openapi.yaml delete /workspaces/{workspaceSlug}/subscription/{subscriptionId}
# Create a workspace
Source: https://thenile.dev/docs/api-reference/workspaces/create-a-workspace
https://global.thenile.dev/openapi.yaml post /workspaces
Creates a workspace for the authenticated developer. A workspace slug is
generated from the workspace name and used as the workspace identifier.
Workspace slugs must be globally unique.
Generated slugs will only include ASCII characters. Spaces and non-ASCII
characters are replaced with hyphens.
# Get current subscription (from DB)
Source: https://thenile.dev/docs/api-reference/workspaces/get-current-subscription-from-db
https://global.thenile.dev/openapi.yaml get /workspaces/{workspaceSlug}/subscription
Get details of the current subscription for the given workspace.
# Get workspaces
Source: https://thenile.dev/docs/api-reference/workspaces/get-workspaces
https://global.thenile.dev/openapi.yaml get /workspaces/{workspaceSlug}
Get details about a specific workspace.
# Get workspaces compute types
Source: https://thenile.dev/docs/api-reference/workspaces/get-workspaces-compute-types
https://global.thenile.dev/openapi.yaml get /workspaces/{workspaceSlug}/compute-types
List dedicated compute types available for a workspace.
# Get workspaces metricscompute
Source: https://thenile.dev/docs/api-reference/workspaces/get-workspaces-metricscompute
https://global.thenile.dev/openapi.yaml get /workspaces/{workspaceSlug}/metrics/compute
List dedicated compute metrics for a workspace.
# List developers with access to a workspace
Source: https://thenile.dev/docs/api-reference/workspaces/list-developers-with-access-to-a-workspace
https://global.thenile.dev/openapi.yaml get /workspaces/{workspaceSlug}/developers
List details about the developers authorized to access the workspace.
# List subscription history (most-recent first)
Source: https://thenile.dev/docs/api-reference/workspaces/list-subscription-history-most-recent-first
https://global.thenile.dev/openapi.yaml get /workspaces/{workspaceSlug}/subscription/history
List all subscription records for the given workspace, most-recent first.
# List workspaces
Source: https://thenile.dev/docs/api-reference/workspaces/list-workspaces
https://global.thenile.dev/openapi.yaml get /workspaces
Lists all workspaces that an authenticated developer is authorized to use.
# Monthly totals by component, from rated lines
Source: https://thenile.dev/docs/api-reference/workspaces/monthly-totals-by-component-from-rated-lines
https://global.thenile.dev/openapi.yaml get /workspaces/{workspaceSlug}/billing/{ym}/totals
# Remove a developer from a workspace
Source: https://thenile.dev/docs/api-reference/workspaces/remove-a-developer-from-a-workspace
https://global.thenile.dev/openapi.yaml delete /workspaces/{workspaceSlug}/developers/{developerId}
Removes a developer from the workspace.
# Start a subscription row
Source: https://thenile.dev/docs/api-reference/workspaces/start-a-subscription-row
https://global.thenile.dev/openapi.yaml post /workspaces/{workspaceSlug}/subscription
# Complete an MFA challenge
Source: https://thenile.dev/docs/auth/api-reference/auth/complete-an-mfa-challenge
https://us-west-2.api.thenile.dev/v2/openapi put /v2/databases/{database}/auth/mfa
Validates the second-factor code that was issued during login or MFA setup. For login challenges, a new session cookie is issued when the supplied code is valid.
# Disable MFA for the current user
Source: https://thenile.dev/docs/auth/api-reference/auth/disable-mfa-for-the-current-user
https://us-west-2.api.thenile.dev/v2/openapi delete /v2/databases/{database}/auth/mfa
Removes the user's active multi-factor credential. When `requireCode` is set or an email method is configured, the request must include a valid MFA code (and token for email) to confirm ownership.
# Get available providers
Source: https://thenile.dev/docs/auth/api-reference/auth/get-available-providers
https://us-west-2.api.thenile.dev/v2/openapi get /v2/databases/{database}/auth/providers
Returns a list of available authentication providers.
# Get CSRF token
Source: https://thenile.dev/docs/auth/api-reference/auth/get-csrf-token
https://us-west-2.api.thenile.dev/v2/openapi get /v2/databases/{database}/auth/csrf
Returns a CSRF token to be used in subsequent requests.
# Get the current session
Source: https://thenile.dev/docs/auth/api-reference/auth/get-the-current-session
https://us-west-2.api.thenile.dev/v2/openapi get /v2/databases/{database}/auth/session
Returns the session object if the user is authenticated.
# Handle provider callback
Source: https://thenile.dev/docs/auth/api-reference/auth/handle-provider-callback
https://us-west-2.api.thenile.dev/v2/openapi post /v2/databases/{database}/auth/callback/{provider}
Handles the callback from an authentication provider.
# Initiate MFA setup
Source: https://thenile.dev/docs/auth/api-reference/auth/initiate-mfa-setup
https://us-west-2.api.thenile.dev/v2/openapi post /v2/databases/{database}/auth/mfa
Begins the multi-factor enrollment flow for the signed-in user by issuing a setup challenge and, when applicable, returning authenticator bootstrap data.
# Refresh session token
Source: https://thenile.dev/docs/auth/api-reference/auth/refresh-session-token
https://us-west-2.api.thenile.dev/v2/openapi post /v2/databases/{database}/auth/session/token
Refreshes the session token to extend the session duration.
# Reset password
Source: https://thenile.dev/docs/auth/api-reference/auth/reset-password
https://us-west-2.api.thenile.dev/v2/openapi post /v2/databases/{database}/auth/reset-password
Sends an email for a user to reset their password
# Resets the password
Source: https://thenile.dev/docs/auth/api-reference/auth/resets-the-password
https://us-west-2.api.thenile.dev/v2/openapi put /v2/databases/{database}/auth/reset-password
Based on a cookie, allows a user to reset their password
# Retrieve password token
Source: https://thenile.dev/docs/auth/api-reference/auth/retrieve-password-token
https://us-west-2.api.thenile.dev/v2/openapi get /v2/databases/{database}/auth/reset-password
Responds to a link (probably in an email) by setting a cookie that allows for a password to be reset
# Sends a verification email
Source: https://thenile.dev/docs/auth/api-reference/auth/sends-a-verification-email
https://us-west-2.api.thenile.dev/v2/openapi post /v2/databases/{database}/auth/verify-email
# Sign in to the application
Source: https://thenile.dev/docs/auth/api-reference/auth/sign-in-to-the-application
https://us-west-2.api.thenile.dev/v2/openapi post /v2/databases/{database}/auth/signin
Authenticates a user and creates a session.
# Sign out of the application
Source: https://thenile.dev/docs/auth/api-reference/auth/sign-out-of-the-application
https://us-west-2.api.thenile.dev/v2/openapi post /v2/databases/{database}/auth/signout
Ends the user session.
# Takes in an email verification token and ensures it is valid
Source: https://thenile.dev/docs/auth/api-reference/auth/takes-in-an-email-verification-token-and-ensures-it-is-valid
https://us-west-2.api.thenile.dev/v2/openapi get /v2/databases/{database}/auth/verify-email
# Accepts a tenant invite
Source: https://thenile.dev/docs/auth/api-reference/tenants/accepts-a-tenant-invite
https://us-west-2.api.thenile.dev/v2/openapi put /v2/databases/{database}/tenants/{tenantId}/invite
Accepts an invite for a user to join a tenant using the token and email. The invite must be valid and not expired.
# creates a tenant
Source: https://thenile.dev/docs/auth/api-reference/tenants/creates-a-tenant
https://us-west-2.api.thenile.dev/v2/openapi post /v2/databases/{database}/tenants
makes a tenant, assigns user to that tenant
# delete a tenant
Source: https://thenile.dev/docs/auth/api-reference/tenants/delete-a-tenant
https://us-west-2.api.thenile.dev/v2/openapi delete /v2/databases/{database}/tenants/{tenantId}
sets a tenant for delete in the database
# delete the current user
Source: https://thenile.dev/docs/auth/api-reference/tenants/delete-the-current-user
https://us-west-2.api.thenile.dev/v2/openapi delete /v2/databases/{database}/me
sets the current user for delete.
# get a tenant
Source: https://thenile.dev/docs/auth/api-reference/tenants/get-a-tenant
https://us-west-2.api.thenile.dev/v2/openapi get /v2/databases/{database}/tenants/{tenantId}
get information about a tenant
# Invite a user to a tenant
Source: https://thenile.dev/docs/auth/api-reference/tenants/invite-a-user-to-a-tenant
https://us-west-2.api.thenile.dev/v2/openapi post /v2/databases/{database}/tenants/{tenantId}/invite
Allows an authenticated tenant member to invite another user via email. The invitee will receive an email and must accept the invitation.
# List pending invites for a tenant
Source: https://thenile.dev/docs/auth/api-reference/tenants/list-pending-invites-for-a-tenant
https://us-west-2.api.thenile.dev/v2/openapi get /v2/databases/{database}/tenants/{tenantId}/invites
Returns all pending invites for a given tenant, accessible by authenticated members of the tenant.
# update a tenant
Source: https://thenile.dev/docs/auth/api-reference/tenants/update-a-tenant
https://us-west-2.api.thenile.dev/v2/openapi put /v2/databases/{database}/tenants/{tenantId}
updates a tenant in the database
# a list of tenant users
Source: https://thenile.dev/docs/auth/api-reference/users/a-list-of-tenant-users
https://us-west-2.api.thenile.dev/v2/openapi get /v2/databases/{database}/tenants/{tenantId}/users
Returns a list of tenant users from the database
# create a new user and assigns them to a tenant
Source: https://thenile.dev/docs/auth/api-reference/users/create-a-new-user-and-assigns-them-to-a-tenant
https://us-west-2.api.thenile.dev/v2/openapi post /v2/databases/{database}/tenants/{tenantId}/users
Creates a brand new user on a tenant
# Creates a user
Source: https://thenile.dev/docs/auth/api-reference/users/creates-a-user
https://us-west-2.api.thenile.dev/v2/openapi post /v2/databases/{database}/users
Adds a brand new user to the database
# Identify the principal
Source: https://thenile.dev/docs/auth/api-reference/users/identify-the-principal
https://us-west-2.api.thenile.dev/v2/openapi get /v2/databases/{database}/me
Returns information about the principal associated with the session provided
# links an existing user to a tenant
Source: https://thenile.dev/docs/auth/api-reference/users/links-an-existing-user-to-a-tenant
https://us-west-2.api.thenile.dev/v2/openapi put /v2/databases/{database}/tenants/{tenantId}/users/{userId}/link
A user that already exists is added to a tenant
# lists tenants of user
Source: https://thenile.dev/docs/auth/api-reference/users/lists-tenants-of-user
https://us-west-2.api.thenile.dev/v2/openapi get /v2/databases/{database}/users/{userId}/tenants
lists the tenants associated with a user
# Signs up a new user
Source: https://thenile.dev/docs/auth/api-reference/users/signs-up-a-new-user
https://us-west-2.api.thenile.dev/v2/openapi post /v2/databases/{database}/signup
Creates a user in the database and then logs them in.
# Unlinks a user from a tenant
Source: https://thenile.dev/docs/auth/api-reference/users/unlinks-a-user-from-a-tenant
https://us-west-2.api.thenile.dev/v2/openapi delete /v2/databases/{database}/tenants/{tenantId}/users/{userId}/link
Marks a user to be deleted from the tenant. It does not remove the user from other tenants or invalidate active sessions.
# update a user
Source: https://thenile.dev/docs/auth/api-reference/users/update-a-user
https://us-west-2.api.thenile.dev/v2/openapi put /v2/databases/{database}/tenants/{tenantId}/users/{userId}
Updates a user, provided the authorized user is in the same tenant as that user
# Update the principal profile
Source: https://thenile.dev/docs/auth/api-reference/users/update-the-principal-profile
https://us-west-2.api.thenile.dev/v2/openapi put /v2/databases/{database}/me
Update the principal associated with the provided session
# Comparison
Source: https://thenile.dev/docs/auth/comparison
There are many auth solutions. A common question is how Nile Auth is different. We have tried to provide our most unbiased opinion about why and when you should consider using Nile Auth. Note that our opinions could be biased, and we encourage our users to do their due diligence.
## Why use Nile Auth over other options?
Nile Auth is a 100% open-source auth solution for B2B apps. It supports multi-tenant workflows, drop-in UI modules, stores user data in Nile’s Postgres database, and you get unlimited active users. You can self host it or use the managed cloud solution.
### vs OSS libraries focused on a specific language or framework
1. Auth as a service. It provides centralized control, helps B2B companies roll out security fixes quickly across all their apps, and gives an easy way to audit
2. Routes auto-generate with Nile Auth. You have to write a lot less backend code.
3. Drop-in B2B components, which makes end-to-end integration possible in a few minutes. Most libraries do not provide this.
4. Tightly integrated to Nile Postgres built-in tables. So, no DB setups are required to bootstrap
5. Multi-language support - A nice benefit is getting auth features across services in multiple languages.
6. With the hosted version, we manage the service and help scale to millions of users across the globe with Nile's Postgres. We offer unlimited active users.
7. If using NextAuth, you do not have any B2B feature support. Nile Auth also supports tenant-level overrides.
8. Unified customer dashboard showing both DB and auth usage with management features
### vs closed-source auth service
1. Not open source.
2. You need to sync user data from a third-party service to your DB. Not required with Nile Auth.
3. We are biased, but Nile Auth UX is superior to most proprietary auth solutions.
4. There is no pricing on active users. For example, Auth0 costs \$1700 for 7500 active users. With with Nile Auth, it is \$0
5. Single customer dashboard showing both DB and auth usage with management features.
6. Routes auto-generate with Nile Auth. You have to write a lot less backend code.
### vs OSS auth service
1. Routes auto-generate with Nile Auth. You have to write a lot less backend code.
2. Drop-in B2B components make end-to-end integration possible in a few minutes, which is not provided by most OSS auth as a service solution.
3. Tightly integrated to Nile Postgres built-in tables. So, no DB setups are required to bootstrap
4. Nile Auth hosted solution offers unlimited active users and only prices for the database.
5. Full support for B2B features, which may not be comprehensive or not fully supported in other solutions like tenant-level overrides.
6. Single customer dashboard showing both DB and auth usage with management features.
### vs build your own auth solution
1. Nile Auth will invest more time and people into security. It may not be the best use of time to build your own auth solution.
2. Nile Auth will be patched proactively, and vulnerabilities will be addressed quickly. It could be hard to do this while focusing on building your product.
3. Nile Auth is open source, and data is stored in your database. There is no lock-in, and you can always access your user data. This has the same benefits as building your own auth solution.
4. With the hosted version, we manage the service and help scale to millions of users across the globe with Nile's Postgres. We offer unlimited active users.
5. A lot of effort is needed to support B2B features. You can leverage Nile Auth for this.
6. Unified customer dashboard showing both DB and auth usage with management features
7. You can self-host Nile Auth, which gives you all the benefits without having to write the code.
# Customize
Source: https://thenile.dev/docs/auth/components/customization
Learn how to style and customize the Nile Auth components
The Nile Auth components are imported from the `@niledatabase/react` package.
They are built with Tailwind CSS and React and are designed to be easily customized.
There are a few ways to customize the components, and the best approach depends on how much customization you need.
In this document we will cover the different approaches and provide examples of how to customize the components.
You will find more details in the documentation for the specific component you are using.
## Nile Auth Default CSS
Nile's react package includes a CSS file that you can use to style the components.
```jsx theme={null}
import '@niledatabase/react/styles.css';
```
This will apply the default styles to all the components and is an easy way to get started
if you just want things to look nice but don't need to match a specific design.
## Styling with Component Props
Some of Nile Auth components are a single element, such as a button. This includes the ``, `` and
all the social login buttons.
These components can be customized using the `className` prop. For example, if you want your signout button to have a different color, you can do the following:
```jsx theme={null}
```
More complex components, such as the ``, ``, `` and `` components,
include child components and text. They can still be customized using the `className` prop, to an extent - for example you can add margins to the form by adding a `className` to the component.
But it isn't the best approach for achieving a specific design.
Some components also have a `buttonText` prop that can be used to change the
text of the button. Check the documentation for the specific component for
more details.
## Theming with CSS Variables
All Nile Auth components use CSS variables for theming. This means that you can override the colors and other styles by setting the CSS variables.
We support the same CSS variables that [Shadcn uses](https://ui.shadcn.com/docs/theming#list-of-variables).
If you are not using Shadcn, you can still use Tailwind CSS to style the components. This requires two steps:
1. Modify Tailwind config to include Nile Auth components and the theme variables.
If you are using a Tailwind config file, add the following to your config:
```js tailwind.config.js [expandable] theme={null}
module.exports = {
content: [
'./src/**/*.{js,jsx,ts,tsx}',
'./node_modules/@niledatabase/react/dist/**/*.js',
],
theme: {
extend: {
colors: {
border: 'hsl(var(--border))',
input: 'hsl(var(--input))',
ring: 'hsl(var(--ring))',
background: 'hsl(var(--background))',
foreground: 'hsl(var(--foreground))',
primary: {
DEFAULT: 'hsl(var(--primary))',
foreground: 'hsl(var(--primary-foreground))',
},
secondary: {
DEFAULT: 'hsl(var(--secondary))',
foreground: 'hsl(var(--secondary-foreground))',
},
destructive: {
DEFAULT: 'hsl(var(--destructive))',
foreground: 'hsl(var(--destructive-foreground))',
},
muted: {
DEFAULT: 'hsl(var(--muted))',
foreground: 'hsl(var(--muted-foreground))',
},
accent: {
DEFAULT: 'hsl(var(--accent))',
foreground: 'hsl(var(--accent-foreground))',
},
popover: {
DEFAULT: 'hsl(var(--popover))',
foreground: 'hsl(var(--popover-foreground))',
},
card: {
DEFAULT: 'hsl(var(--card))',
foreground: 'hsl(var(--card-foreground))',
},
},
borderRadius: {
lg: 'var(--radius)',
md: 'calc(var(--radius) - 2px)',
sm: 'calc(var(--radius) - 4px)',
},
fontFamily: {
sans: ['var(--font-sans)', ...fontFamily.sans],
roboto: ['Roboto', 'sans-serif'],
},
},
},
plugins: [],
};
```
If you are using Tailwind v4 without a config file, add the following to your `globals.css` file:
```css global.css [expandable] theme={null}
@source '../../node_modules/@niledatabase/react/**/*.{html,js,jsx,ts,tsx}';
@theme {
--theme-background-color: hsl(var(--background));
--theme-foreground-color: hsl(var(--foreground));
--radius: 0.5rem;
--color-border: hsl(var(--border));
--color-input: hsl(var(--input));
--color-ring: hsl(var(--ring));
--color-background: hsl(var(--background));
--color-foreground: hsl(var(--foreground));
--color-primary: hsl(var(--primary));
--color-primary-foreground: hsl(var(--primary-foreground));
--color-secondary: hsl(var(--secondary));
--color-secondary-foreground: hsl(var(--secondary-foreground));
--color-destructive: hsl(var(--destructive));
--color-destructive-foreground: hsl(var(--destructive-foreground));
--color-muted: hsl(var(--muted));
--color-muted-foreground: hsl(var(--muted-foreground));
--color-accent: hsl(var(--accent));
--color-accent-foreground: hsl(var(--accent-foreground));
--color-popover: hsl(var(--popover));
--color-popover-foreground: hsl(var(--popover-foreground));
--color-card: hsl(var(--card));
--color-card-foreground: hsl(var(--card-foreground));
--radius-sm: calc(var(--radius) - 4px);
--radius-md: calc(var(--radius) - 2px);
--radius-lg: var(--radius);
--font-family-poppins: var(--font-poppins);
--font-family-inter: var(--font-inter);
}
```
2. Add the theme variables to the `:root` selector in the `globals.css` file. You can use [shadcn theme generator](https://ui.shadcn.com/themes) to get
a nice theme for your app. In the example below, I picked a purple theme.
```css [expandable] theme={null}
@layer base {
:root {
--background: 276 100% 95%;
--foreground: 276 5% 10%;
--card: 276 50% 90%;
--card-foreground: 276 5% 15%;
--popover: 276 100% 95%;
--popover-foreground: 276 100% 10%;
--primary: 276 85% 48%;
--primary-foreground: 0 0% 100%;
--secondary: 276 30% 70%;
--secondary-foreground: 0 0% 0%;
--muted: 238 30% 85%;
--muted-foreground: 276 5% 35%;
--accent: 238 30% 80%;
--accent-foreground: 276 5% 15%;
--destructive: 0 100% 30%;
--destructive-foreground: 276 5% 90%;
--border: 276 30% 50%;
--input: 276 30% 26%;
--ring: 276 85% 48%;
--radius: 0.5rem;
}
.dark {
--background: 276 50% 10%;
--foreground: 276 5% 90%;
--card: 276 50% 10%;
--card-foreground: 276 5% 90%;
--popover: 276 50% 5%;
--popover-foreground: 276 5% 90%;
--primary: 276 85% 48%;
--primary-foreground: 0 0% 100%;
--secondary: 276 30% 20%;
--secondary-foreground: 0 0% 100%;
--muted: 238 30% 25%;
--muted-foreground: 276 5% 60%;
--accent: 238 30% 25%;
--accent-foreground: 276 5% 90%;
--destructive: 0 100% 30%;
--destructive-foreground: 276 5% 90%;
--border: 276 30% 26%;
--input: 276 30% 26%;
--ring: 276 85% 48%;
--radius: 0.5rem;
}
}
```
## Additional customizations
If you need more customization, if you are not using Tailwind CSS, or if you have your own UI component library,
you can use Nile Auth React hooks with any components you want and any styles you want.
For example, to use Nile's SignIn hook with your own button, you can do the following:
```jsx theme={null}
import { useSignIn } from '@niledatabase/react';
const SignInComponent = () => {
const signIn = useSignIn({
beforeMutate: (data) => ({ ...data, extraParam: 'value' }),
onSuccess: () => console.log('Login successful'),
onError: (error) => console.error('Login failed', error),
callbackUrl: '/dashboard',
});
const handleLogin = () => {
signIn({ email: 'user@example.com', password: 'securepassword' });
};
return Sign In ;
};
```
This snippet uses the `useSignIn` hook to sign in a user and the `beforeMutate` option to add an extra parameter to the sign in request.
Note how customizable the hook is - you can add extra parameters, handle the success and error cases, and redirect the user after login.
Individual component pages also include the relevant hooks and code snippets you need to use the component in your own code.
# Multi-factor
Source: https://thenile.dev/docs/auth/components/multifactor
Nile Auth React components for MFA setup, challenge, and disable flows
Client-side MFA UI powered by `@niledatabase/react`. These components pair with `useMultiFactor` (and the low-level `mfa` helper) to enroll authenticator or email factors, verify challenges, and disable MFA when needed. The `User` object (obtained via `useSession`) will include a `multiFactor` property if MFA is enabled for the user.
## Authenticator setup
` ` renders the otpauth QR, recovery codes, and a verification form. Pair it with `useMultiFactor` to start setup and render the payload.
```tsx theme={null}
import { useMultiFactor, MultiFactorAuthenticator } from '@niledatabase/react';
export function AuthenticatorSetupCard() {
const { setup, loading, startSetup, errorType } = useMultiFactor({
method: 'authenticator',
currentMethod: null,
});
return (
{loading ? 'Starting...' : 'Enable authenticator MFA'}
{setup?.method === 'authenticator' && setup.scope === 'setup' ? (
console.error('MFA error', msg)}
onSuccess={(scope) => scope === 'setup' && window.location.reload()}
/>
) : null}
{errorType ? Unable to start MFA ({errorType})
: null}
);
}
```
## Email setup
` ` shows masked email messaging and a verification form. Pair it with `useMultiFactor` to start setup and render the payload.
```tsx theme={null}
import { useMultiFactor, MultiFactorEmail } from '@niledatabase/react';
export function EmailSetupCard() {
const { setup, loading, startSetup, errorType } = useMultiFactor({
method: 'email',
currentMethod: null,
});
return (
{loading ? 'Starting...' : 'Enable email MFA'}
{setup?.method === 'email' && setup.scope === 'setup' ? (
scope === 'setup' && window.location.reload()}
/>
) : null}
{errorType ? Unable to start MFA ({errorType})
: null}
);
}
```
## Challenge content (sign-in or disable)
` ` verifies a code during sign-in, setup verification, or disable flows. Feed it the `setup` payload returned from `useMultiFactor` (or the `/auth/mfa` API) and handle success.
```tsx theme={null}
import { useMultiFactor, MultiFactorChallenge } from '@niledatabase/react';
export function AuthenticatorChallenge({ existingToken }: { existingToken?: string }) {
const { setup, startDisable } = useMultiFactor({
method: 'authenticator',
currentMethod: 'authenticator',
});
// Example: startDisable() would populate `setup` with a challenge payload when confirmation is required.
return (
Disable MFA
{setup && setup.scope === 'challenge' ? (
window.location.replace('/app')}
/>
) : null}
);
}
```
## Hooks
### `useMultiFactor(options)`
Manages MFA setup and disable flows and provides the payloads rendered by the components above.
| Name | Type | Default | Description |
| --------------------- | ---------------------------------------------------------- | --------------------------------- | ----------------------------------------------------------- |
| `method` | `'authenticator' \| 'email'` | *(required)* | MFA mechanism to enable or disable. |
| `currentMethod` | `'authenticator' \| 'email' \| null` | `null` | Currently enrolled method; blocks switching until disabled. |
| `onRedirect` | `(url: string) => void` | `window.location.assign` | Handle `{ url }` redirect responses (`ChallengeRedirect`). |
| `onChallengeRedirect` | `(params: { token; method; scope; destination? }) => void` | Internal `/mfa/prompt` navigation | Override the default challenge redirect builder. |
Returns `{ setup, loading, errorType, startSetup, startDisable }`.
* `setup`: MFA payload for rendering components. Authenticator shape: `{ method: 'authenticator'; token; scope; otpauthUrl?; secret?; recoveryKeys? }`. Email shape: `{ method: 'email'; token; scope; maskedEmail? }`.
* `startSetup()`: begins enrollment (POST `/auth/mfa`); `setup.scope` can be `"setup"` or `"challenge"` when verification is required.
* `startDisable()`: starts removal for the given method. If verification is required, `setup.scope` will be `"challenge"`.
* `errorType`: one of `setup`, `disable`, `parseSetup`, `parseDisable`, or `null`.
### `mfa(options)` (low-level helper)
Wrapper around `/auth/mfa` from `@niledatabase/client`. Use it for custom prompts or headless flows; the React components call this under the hood.
* `POST` `{ method, scope: 'setup' }` → returns a setup payload.
* `PUT` `{ token, code, method, scope }` → verifies a challenge; returns `{ ok: true; scope; recoveryCodesRemaining? }`.
* `DELETE` `{ token?, code?, method, remove: true }` → disables MFA; may require a valid code/token depending on backend settings.
* Redirects surface as `{ url: string }` (`ChallengeRedirect`); parse `error` from the query string for messaging.
**Notes**
* Codes are 6 digits for authenticator/email verification; recovery codes are string tokens issued during setup.
* Tokens expire; expect 410 for stale tokens and 404 for missing challenges.
* Email MFA may require the same token for verification and disable flows.
## Related Components
* [MFA Concepts](/auth/concepts/multifactor)
* [Sign In](/auth/components/signin)
* [Sign Out](/auth/components/signout)
* [User Profile](/auth/components/user)
```
```
# Organization
Source: https://thenile.dev/docs/auth/components/organization
Learn how to use the Nile Auth Organization component
# Tenant Selector
The `TenantSelector` component provides an interactive way to manage and switch between tenant organizations within an application. It leverages React Query for fetching tenant data and includes functionality to create new tenants.
***
## Features
* **Tenant Selection**: Users can switch between available tenants using a dropdown.
* **Tenant Creation**: If no tenants exist, users can create a new organization.
* **Persistent Selection**: Selected tenants persist via cookies.
* **Server Communication**: Fetches tenant data from an API endpoint.
***
## Installation
Ensure you have the necessary dependencies installed:
```sh theme={null}
npm install @niledatabase/react
```
***
## Usage
```tsx theme={null}
import TenantSelector from '@niledatabase/react';
export default function App() {
return ;
}
```
***
#### Props
| Name | Type | Default | Description |
| ----------- | ------------- | ------- | --------------------------------- |
| `client` | `QueryClient` | `null` | React Query client instance. |
| `baseUrl` | `string` | `''` | Base API URL for tenant requests. |
| `className` | `string` | `''` | Additional CSS classes. |
***
### `useTenants`
A hook to fetch tenant data.
```ts theme={null}
const { data: tenants, isLoading, refetch } = useTenants();
```
Returns:
* `tenants`: List of available tenants.
* `isLoading`: Whether data is loading.
* `refetch()`: Function to refresh data.
***
### `useTenantId`
A hook for managing the currently selected tenant.
```ts theme={null}
const [tenantId, setTenant] = useTenantId();
```
Returns:
* `tenantId`: The currently selected tenant ID.
* `setTenant(id: string)`: Function to change the active tenant.
***
### `CreateTenant`
A component for creating a new organization.
```tsx theme={null}
console.log('Created:', tenant)}
trigger={Create Tenant }
/>
```
#### Props
| Name | Type | Description |
| ----------- | -------------------------- | --------------------------------- |
| `trigger` | `React.ReactElement` | Element to trigger modal. |
| `onSuccess` | `(tenant: Tenant) => void` | Callback when tenant is created. |
| `onError` | `(error: Error) => void` | Callback when creation fails. |
| `fetchUrl` | `string` | API endpoint for tenant creation. |
***
## Behavior
1. **Tenant Selection**: Users can select an existing tenant from a dropdown.
2. **Tenant Creation**: If no tenant exists, users can create one.
3. **Persistent State**: Selected tenant is saved via cookies.
4. **Automatic Data Fetching**: Tenant data is fetched on component mount.
***
## Example UI
* If the user has tenants, they can switch between them.
* If the user has no tenants, they are prompted to create one.
* Clicking "Create Tenant" opens a form to enter an organization name.
## Theming
The component is styled using tailwind. It can be customized using the `className` and `buttonText` props,
or using Tailwind CSS theme variables. You can find more details and full examples in the [customization doc](/auth/components/customization).
In addition, the hooks documented for each component can be used if you'd like to use Nile Auth with your own components.
If needed, you can peek at the [source code of the component](https://github.com/niledatabase/nile-js/tree/main/packages/react/src) and use it as a template to create your own component for maximum customization.
## Related Topics
* [User Management](/auth/concepts/users)
* [Tenants](/auth/concepts/tenants)
* [User Component](/auth/components/user)
# Sign In
Source: https://thenile.dev/docs/auth/components/signin
Learn how to use the Nile Auth Sign In component
Users can sign up and be entered into the database via email + password. A JWT is used (which has a default expiry in 30 days). For Single Sign On, sessions are kept in the database in the `auth.session` table.
`@tanstack/react-query` and `react-hook-form` are used to do much of the heavy lifting in the component. The component is styled via tailwind.
## Installation
```bash theme={null}
npm install @niledatabase/react @niledatabase/client
```
## Email + Password Sign In
Uses simple ` ` and ` ` fields, along with the `useSignIn()` hook to sign a user in, if they exist. In most cases, you want to supply a `callbackUrl`, which is where the user will be redirected upon successful sign in. Error handling works the same as it does in ` `
```jsx theme={null}
import { SignInForm } from "@niledatabase/react";
export default function SignUpPage() {
return
}
```
## Email Only Sign In
With an SMTP provider configured on your database, users can supply their emails and be sent a link for sign in. The users enters their email, which causes the auth service to create a token (saved to the auth.verification\_tokens table in your database), and send it in an email. The token is valid for 4 hours by default.
Once the user clicks the link, they will exchange their token for a session and be redirected to whichever page was provided in the `callbackUrl`
```jsx theme={null}
import { EmailSignIn } from "@niledatabase/react";
export default function SignUpPage() {
return
}
```
## Hooks
### useSignIn
The `useSignIn` hook provides a way to authenticate users using credential-based sign-in within a React application. It utilizes React Query's `useMutation` to handle authentication requests and supports optional lifecycle callbacks for customization.
#### Installation
```sh theme={null}
npm install @niledatabase/react @niledatabase/client
```
#### Usage
```tsx theme={null}
import { useSignIn } from '@niledatabase/react';
const SignInComponent = () => {
const signIn = useSignIn({
beforeMutate: (data) => ({ ...data, extraParam: 'value' }),
onSuccess: () => console.log('Login successful'),
onError: (error) => console.error('Login failed', error),
callbackUrl: '/dashboard',
});
const handleLogin = () => {
signIn({ email: 'user@example.com', password: 'securepassword' });
};
return Sign In ;
};
```
#### API
##### `useSignIn(params?: Props): (data: LoginInfo) => void`
##### Parameters
| Name | Type | Description |
| -------- | -------------------- | --------------------- |
| `params` | `Props` *(optional)* | Configuration options |
##### Returns
A function that accepts `LoginInfo` containing user credentials to initiate the sign-in process.
#### Props
| Name | Type | Description |
| -------------- | ------------------------------------------------ | --------------------------------------------------------------------------------------------- |
| `callbackUrl` | `string` *(recommended)* | The URL to redirect to upon successful login. |
| `beforeMutate` | `(data: any) => any` *(optional)* | Optional function executed before the mutation occurs, allowing modification of request data. |
| `onSuccess` | `(data: LoginSuccess) => void` *(optional)* | Callback function triggered on a successful login. |
| `onError` | `(error: Error, data: any) => void` *(optional)* | Callback function triggered if the login fails. |
| `client` | `QueryClient` *(optional)* | React Query client instance. |
##### LoginInfo Type
```ts theme={null}
export type LoginInfo = {
email: string;
password: string;
};
```
##### Example with Error Handling
```tsx theme={null}
const signIn = useSignIn({
onSuccess={async (res) => {
if (res.ok) {
// something good happened
} else {
const error = await res.text();
console.error(error);
}
}}
onError={(e)=> {
console.error('Catastrophic failure 😭', e)
}}
});
signIn({ email: "user@example.com", password: "wrongpassword" });
```
* Ensure that authentication providers are properly configured in your Next.js app.
* Use `beforeMutate` to modify request payloads before submission.
* The `callbackUrl` parameter determines the redirection URL post-login.
## Theming
The component is styled using tailwind. It can be customized using the `className` and `buttonText` props,
or using Tailwind CSS theme variables. You can find more details and full examples in the [customization doc](/auth/components/customization).
In addition, the hooks documented for each component can be used if you'd like to use Nile Auth with your own components.
If needed, you can peek at the [source code of the component](https://github.com/niledatabase/nile-js/tree/main/packages/react/src) and use it as a template to create your own component for maximum customization.
## Related Components
* [Single Sign On](/auth/singlesignon)
* [Sign Out](/auth/components/signout)
* [User Profile](/auth/components/user)
# Sign Out
Source: https://thenile.dev/docs/auth/components/signout
Learn how to use the Nile Auth Sign Out component
## Overview
The `SignOutButton` component provides an interactive button for signing users out of your application. It supports optional redirect behavior, a custom callback URL, and configurable button text.
## Installation
```sh theme={null}
npm install @niledatabase/react @niledatabase/client
```
## Usage
```jsx theme={null}
import { SignOutButton } from "@niledatabase/react";
const MyComponent = () => {
return ;
};
```
### `SignOutButton` Props
| Name | Type | Default | Description |
| ------------- | --------- | ---------------------- | ------------------------------------------------------------------- |
| `redirect` | `boolean` | `true` | Determines whether the user should be redirected after signing out. |
| `callbackUrl` | `string` | `window.location.href` | The URL to redirect to after signing out. |
| `buttonText` | `string` | `'Sign out'` | Custom text for the sign-out button. |
## Example with No Redirect
```tsx theme={null}
```
## Internal Functionality
The `SignOutButton` component utilizes the `signOut` function to trigger a sign-out request.
```tsx theme={null}
signOut({ callbackUrl: '/sign-in' }); // Redirects user to the sign-in page
signOut({ redirect: false }); // Logs out without redirecting
```
### `signOut` Function
The `signOut` function makes a `POST` request to the sign-out endpoint, optionally taking:
* `callbackUrl`: A URL to redirect the user after logging out.
* `redirect`: A boolean indicating whether to reload the page post-logout.
## Theming
The component is styled using tailwind. It can be customized using the `className` and `buttonText` props,
or using Tailwind CSS theme variables. You can find more details and full examples in the [customization doc](/auth/components/customization).
In addition, the hooks documented for each component can be used if you'd like to use Nile Auth with your own components.
If needed, you can peek at the [source code of the component](https://github.com/niledatabase/nile-js/tree/main/packages/react/src) and use it as a template to create your own component for maximum customization.
## Related Components
* [Sign In](/auth/components/signin)
* [Sign Up](/auth/components/signup)
* [User Profile](/auth/components/user)
# Sign Up
Source: https://thenile.dev/docs/auth/components/signup
Learn how to use the Nile Auth Sign Up component
## Email + password
The ` ` will create a new user in your database and create a session for them. Uses simple ` ` and ` ` fields, along with the `useSignUp()` hook. In most cases, you want to supply a `callbackUrl`, which is where the user will be redirected upon successful sign up and sign in.
```jsx theme={null}
import { SignUpForm } from "@niledatabase/react";
export default function SignUpPage() {
return
}
```
#### Error handling
In the case of errors from the API, or you want to do something more custom with the response, you can use the `onError` and `onSuccess` callbacks.
Note that `redirect` is set to the current page by default, so you may want to set it to `false` if you want
to handle the redirect yourself - especially if you want to display a custom error message.
```jsx theme={null}
export default function SignUpPage() {
const [error, setError] = error('')
return (
{error && (
{error}
)}
{
setError(e.message);
}}
onSuccess={(data) => {
console.log(data); // do something with the data
router.push("/"); // redirect to a new page
}}
/>
)
}
```
#### Custom sign up form
```jsx theme={null}
import React from 'react';
import { useForm } from 'react-hook-form';
import { QueryClient, QueryClientProvider } from '@tanstack/react-query';
import { Button } from '../../components/ui/button';
import { Input } from '../../components/ui/input';
import {
Email,
Form,
FormControl,
FormDescription,
FormField,
FormItem,
FormLabel,
FormMessage,
Password,
} from '../../components/ui/form';
import { useSignUp } from '@niledatabase/react';
function SignUpCustom() {
const signUp = useSignUp({
onSuccess: () => {
// nothing to do here
},
});
const form = useForm({
defaultValues: {
givenName: '',
name: '',
familyName: '',
picture: '',
email: '',
password: '',
newTenantName: '',
},
});
return (
);
}
const queryClient = new QueryClient();
export function CustomSignUpForm() {
return (
);
}
```
## Related Components
* [Sign In](/auth/components/signin)
* [Sign Out](/auth/components/signout)
* [User Profile](/auth/components/user)
```
```
# Users
Source: https://thenile.dev/docs/auth/components/user
Learn how to use the Nile Auth User component
## Overview
The `UserInfo` component displays a user's profile information, including their name, email, and profile picture. It supports fetching user data dynamically if no user is provided initially.
## Installation
Ensure you have the required dependencies installed:
```sh theme={null}
npm install @niledatabase/react @niledatabase/client
```
```jsx theme={null}
import { UserInfo } from '@niledatabase/react';
export default function ProfilePage() {
return ;
}
```
## Props
| Name | Type | Description |
| --------------------------- | --------------------------------- | ------------------------------------------------------------------------- |
| `user` | `User \| null \| undefined` | The user object, if already available. |
| `fetchUrl` | `string` *(optional)* | API endpoint to fetch user data if not provided. Defaults to `"/api/me"`. |
| `profilePicturePlaceholder` | `React.ReactElement` *(optional)* | Placeholder component for the profile picture if the user has none. |
## User Type
The `User` type includes:
```ts theme={null}
export type User = {
name?: string;
givenName?: string;
familyName?: string;
email: string;
emailVerified?: boolean;
picture?: string;
created: string;
};
```
## Features
* **Displays user information**: Name, email, profile picture, and account creation date.
* **Email verification badge**: Shows a verification badge if the email is verified.
* **Auto-fetches user data**: If `user` is not provided, it fetches data from `fetchUrl`.
* **Customizable profile picture placeholder**: Supports a fallback element.
## Example with Preloaded User Data
```tsx theme={null}
import { UserInfo } from '@niledatabase/react';
const user = {
name: 'John Doe',
email: 'johndoe@example.com',
emailVerified: true,
picture: 'https://example.com/avatar.jpg',
created: '2023-01-01T12:00:00Z',
};
export default function ProfilePage() {
return ;
}
```
## Example with Fallback Image
```tsx theme={null}
import UserInfo from '@niledatabase/react';
import { CircleUserRound } from 'lucide-react';
export default function ProfilePage() {
return (
}
/>
);
}
```
* If `user` is provided, the component will not make a fetch request.
* Profile pictures are displayed if available; otherwise, a random gradient is used as a placeholder.
* The component ensures proper accessibility by setting the `alt` attribute for images.
## Theming
The component is styled using tailwind. It can be customized using the `className` and `buttonText` props,
or using Tailwind CSS theme variables. You can find more details and full examples in the [customization doc](/auth/components/customization).
In addition, the hooks documented for each component can be used if you'd like to use Nile Auth with your own components.
If needed, you can peek at the [source code of the component](https://github.com/niledatabase/nile-js/tree/main/packages/react/src) and use it as a template to create your own component for maximum customization.
## Related Components
* [Sign In](/auth/components/signin)
* [Sign Up](/auth/components/signup)
* [Organization](/auth/components/organization)
# Account Linking
Source: https://thenile.dev/docs/auth/concepts/acclinking
Understanding account linking in Nile Auth
## Overview
Account linking allows users to authenticate with multiple SSO providers to the same user.
This is automatically enabled in Nile Auth, and will be used if you have multiple SSO providers configured
or if you choose to support both email/password and SSO authentication.
Combining SSO and email/password authentication requires the email address to
be verified. You can set up [email provider and templates](/auth/email) in
Nile console.
## How it works
When a user first signs up, no matter what authentication method they use, they are assigned a unique user ID that is linked to their email address.
You can see this in the `users.users` table:
```sql theme={null}
SELECT * FROM users.users;
```
If the user then signs in with a different SSO provider, and the email address is the same, the new authentication method will be linked to the same user ID.
You can see that the user ID has multiple credentials in the `users.credentials` table:
```sql theme={null}
SELECT * FROM users.credentials where user_id = '...';
```
The user details in the `users.users` table are not affected by account linking. This means that the user will still have the same user ID,
the same email address, and the same user details such as `first_name`, `last_name`, and `picture`. The only thing that changes is the addition of
the new authentication method in the `users.credentials` table.
## Understanding Account Linking Scenarios
There are three scenarios that can occur when a user signs up with multiple authentication methods, and in order to maintain the security of the system,
the behavior is slightly different for each scenario:
### User signs up with multiple SSO providers
If you have multiple SSO providers configured, and a user has the same email address across all of them,
the user will be linked to the same user ID. This is the simplest and most secure scenario.
### User signs up with email/password and then signs in with SSO
If a user signs up with email/password and then signs in with a SSO provider, the SSO provider will be linked to the same user ID.
However, unless the email address is **verified** in Nile Auth, the user will not be able to sign in to their email/password account after linking.
Why? Because the email address is not verified, it is possible that a malicious actor created the same email address, and could potentially
hijack the account. In order to prevent this, we require the email address to be verified before the user can sign in to their email/password account
after linking. You can use our [email verification feature](/auth/email/verification) to automatically send developers an email to verify their
email address when they sign up. Or alternatively, you can manually verify any email address in the Nile console.
For this reason, we strongly recommend that you verify email addresses in your production application.
We support unverified email addresses, but we recommend limiting this to development environments.
### User signs up with SSO and then signs in with email/password
If a user signs up with a SSO provider and then wants to add email/password authentication to their account, they can do this by first signing in
with their SSO provider, and then use the reset-password form to add email/password authentication to their account.
This will create a new email/password credential for the user, and link it to the same user ID.
# Built-in Tables
Source: https://thenile.dev/docs/auth/concepts/builtintables
Understanding built-in database tables in Nile Auth
## User Tables
### `users.users`
```sql theme={null}
Column | Type | Collation | Nullable | Default
----------------+-----------------------------+-----------+----------+---------------------------
id | uuid | | not null | public.uuid_generate_v7()
created | timestamp without time zone | | not null | LOCALTIMESTAMP
updated | timestamp without time zone | | not null | LOCALTIMESTAMP
deleted | timestamp without time zone | | |
name | text | | |
family_name | text | | |
given_name | text | | |
email | text | | |
picture | text | | |
email_verified | timestamp without time zone | | |
Indexes:
"users_pkey" PRIMARY KEY, btree (id)
"users_email_key" UNIQUE, btree (email) WHERE deleted IS NULL
Referenced by:
TABLE "auth.credentials" CONSTRAINT "credentials_user_id_fkey" FOREIGN KEY (user_id) REFERENCES users(id)
TABLE "auth.oidc_auth_attempts" CONSTRAINT "oidc_auth_attempts_user_id_fkey" FOREIGN KEY (user_id) REFERENCES users(id)
TABLE "auth.sessions" CONSTRAINT "sessions_userId_fkey" FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE
```
### `users.tenant_users`
```sql theme={null}
Column | Type | Collation | Nullable | Default
-----------+-----------------------------+-----------+----------+----------------
tenant_id | uuid | | not null |
user_id | uuid | | not null |
created | timestamp without time zone | | not null | LOCALTIMESTAMP
updated | timestamp without time zone | | not null | LOCALTIMESTAMP
deleted | timestamp without time zone | | |
roles | text[] | | |
email | text | | |
Indexes:
"tenant_users_pkey" PRIMARY KEY, btree (tenant_id, user_id)
Foreign-key constraints:
"tenant_users_tenant_id_fkey" FOREIGN KEY (tenant_id) REFERENCES tenants(id)
Referenced by:
TABLE "auth.tenant_oidc_auth_attempts" CONSTRAINT "tenant_oidc_auth_attempts_tenant_id_user_id_fkey" FOREIGN KEY (tenant_id, user_id) REFERENCES tenant_users(tenant_id, user_id)
```
## Authentication Tables
### `auth.sessions`
```sql theme={null}
Column | Type | Collation | Nullable | Default
---------------+--------------------------+-----------+----------+-------------------
id | uuid | | not null | gen_random_uuid()
expires_at | timestamp with time zone | | not null |
session_token | text | | not null |
user_id | uuid | | not null |
Indexes:
"sessions_pkey" PRIMARY KEY, btree (id)
"sessiontoken_unique" UNIQUE CONSTRAINT, btree (session_token)
Foreign-key constraints:
"sessions_userId_fkey" FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE
```
### `auth.credentials`
```sql theme={null}
Column | Type | Collation | Nullable | Default
---------------------+------------------------------+-----------+----------+---------------------------
id | uuid | | not null | public.uuid_generate_v7()
user_id | uuid | | not null |
created | timestamp without time zone | | not null | LOCALTIMESTAMP
updated | timestamp without time zone | | not null | LOCALTIMESTAMP
deleted | timestamp without time zone | | |
method | public.authentication_method | | not null |
provider | text | | not null | 'nile'::text
payload | jsonb | | |
provider_account_id | text | | |
Indexes:
"credentials_pkey" PRIMARY KEY, btree (id)
Foreign-key constraints:
"credentials_user_id_fkey" FOREIGN KEY (user_id) REFERENCES users(id)
```
### `auth.verification_tokens`
```sql theme={null}
Column | Type | Collation | Nullable | Default
------------+-----------------------------+-----------+----------+---------------------------------------
identifier | text | | not null |
token | text | | not null |
expires | timestamp without time zone | | not null | LOCALTIMESTAMP + '04:00:00'::interval
Indexes:
"verification_tokens_pkey" PRIMARY KEY, btree (identifier)
"token_identifier_unique" UNIQUE CONSTRAINT, btree (token, identifier)
```
### `auth.invites`
```sql theme={null}
Column | Type | Collation | Nullable | Default
------------+--------------------------+-----------+----------+-------------------
id | uuid | | not null | gen_random_uuid()
tenant_id | uuid | | not null |
token | text | | not null |
identifier | text | | not null |
roles | text | | |
created_by | uuid | | |
expires | timestamp with time zone | | | LOCALTIMESTAMP + '7 days'::interval
Indexes:
"auth_invites_pkey" PRIMARY KEY, btree (tenant_id, id)
"unique_invite_per_email_per_tenant" UNIQUE, btree (tenant_id, identifier)
```
## OIDC Provider tables
### `auth.oidc_providers`
```sql theme={null}
Column | Type | Collation | Nullable | Default
--------------+-----------------------------+-----------+----------+---------------------------
id | uuid | | not null | public.uuid_generate_v7()
created | timestamp without time zone | | not null | LOCALTIMESTAMP
updated | timestamp without time zone | | not null | LOCALTIMESTAMP
deleted | timestamp without time zone | | |
name | text | | not null |
auth_type | public.provider_auth_type | | not null |
enabled | boolean | | not null | false
config_url | text | | |
redirect_url | text | | |
config | jsonb | | |
ttl_sec | integer | | not null | 3600
Indexes:
"oidc_providers_pkey" PRIMARY KEY, btree (id)
"oidc_providers_name_key" UNIQUE CONSTRAINT, btree (name)
Referenced by:
TABLE "auth.oidc_relying_parties" CONSTRAINT "oidc_relying_parties_provider_fkey" FOREIGN KEY (provider) REFERENCES auth.oidc_providers(id)
```
### `auth.oidc_relying_parties`
```sql theme={null}
Column | Type | Collation | Nullable | Default
---------------+-----------------------------+-----------+----------+---------------------------
id | uuid | | not null | public.uuid_generate_v7()
provider | uuid | | not null |
created | timestamp without time zone | | not null | LOCALTIMESTAMP
updated | timestamp without time zone | | not null | LOCALTIMESTAMP
deleted | timestamp without time zone | | |
client_id | text | | not null |
client_secret | text | | not null |
enabled | boolean | | not null | true
Indexes:
"oidc_relying_parties_pkey" PRIMARY KEY, btree (id)
"oidc_relying_parties_provider_key" UNIQUE, btree (provider, (deleted IS NULL))
Foreign-key constraints:
"oidc_relying_parties_provider_fkey" FOREIGN KEY (provider) REFERENCES auth.oidc_providers(id)
Referenced by:
TABLE "auth.oidc_auth_attempts" CONSTRAINT "oidc_auth_attempts_relying_party_fkey" FOREIGN KEY (relying_party) REFERENCES auth.oidc_relying_parties(id)
```
## Tenant override tables
### `auth.tenant_oidc_relying_parties`
```sql theme={null}
Column | Type | Collation | Nullable | Default
---------------+-----------------------------+-----------+----------+---------------------------
id | uuid | | not null | public.uuid_generate_v7()
tenant_id | uuid | | not null |
provider_name | text | | not null |
created | timestamp without time zone | | not null | LOCALTIMESTAMP
updated | timestamp without time zone | | not null | LOCALTIMESTAMP
deleted | timestamp without time zone | | |
enabled | boolean | | not null | true
config_url | text | | not null |
config | jsonb | | |
ttl_sec | integer | | not null | 3600
client_id | text | | not null |
client_secret | text | | not null |
domains | text[] | | not null |
Indexes:
"tenant_oidc_relying_parties_pkey" PRIMARY KEY, btree (id, tenant_id)
"tenant_oidc_provider_key" UNIQUE, btree (tenant_id, provider_name, (deleted IS NULL))
Foreign-key constraints:
"tenant_oidc_relying_parties_tenant_id_fkey" FOREIGN KEY (tenant_id) REFERENCES tenants(id)
Referenced by:
TABLE "auth.tenant_oidc_auth_attempts" CONSTRAINT "tenant_oidc_auth_attempts_registration_id_tenant_id_fkey" FOREIGN KEY (registration_id, tenant_id) REFERENCES auth.tenant_oidc_relying_parties(id, tenant_id)
```
* [Tenants](/auth/concepts/tenants)
* [Users](/auth/concepts/users)
* [Per-tenant Overrides](/auth/concepts/per-tenantoverrides)
# Cookies
Source: https://thenile.dev/docs/auth/concepts/cookies
Understanding cookie usage in Nile Auth
## What are Cookies?
A cookie is a small piece of data that is sent from a website and stored in the user's browser. Cookies are often used to store session data, preferences, or authentication information, allowing websites to remember users between sessions. Each time the user revisits the site, the browser sends the stored cookies back to the server to maintain a persistent session or store user-specific data.
## Nile Auth Cookies
Nile Auth uses cookies to maintain user sessions and facilitate secure authentication flows.
Here are the key cookies used by Nile Auth:
### Session Token Cookie
* **Cookie Name**: `nile.session-token`
* **Purpose**: This cookie stores the session token (usually a JWT or a session ID) to maintain the user's authenticated session across multiple requests.
* **How it Works**: When a user logs in successfully, the session token is stored in this cookie. This token is then sent automatically with each request to the server, allowing the backend to authenticate the user and retrieve session data.
* **Options**:
* `httpOnly`: Ensures the cookie is inaccessible to JavaScript, preventing XSS attacks.
* `secure`: Ensures the cookie is only sent over HTTPS.
* `sameSite: 'lax'`: Prevents the cookie from being sent with cross-site requests, but allows it for top-level navigation.
* `path: '/'`: Makes the cookie available across the entire site.
### Callback URL Cookie
* **Cookie Name**: `nile.callback-url`
* **Purpose**: This cookie stores the URL the user should be redirected to after completing an authentication flow (such as logging in via OAuth).
* **How it Works**: Before redirecting a user to an external provider for authentication, the `callbackUrl` is stored in this cookie. Once authentication is successful, the user is redirected back to this URL.
* **Options**:
* `sameSite: 'lax'`: This ensures the cookie is only sent when making same-site requests, improving security.
* `secure`: Ensures the cookie is only sent over HTTPS connections.
* `path: '/'`: Ensures the cookie is available across the entire site.
### CSRF Token Cookie
* **Cookie Name**: `nile.csrf-token`
* **Purpose**: This cookie stores a CSRF token to protect against cross-site request forgery attacks.
* **How it Works**: The CSRF token is generated during the login or authorization process and stored in this cookie. It must then be included in the headers of state-changing requests (e.g., form submissions, API calls). The server checks the validity of the token to ensure the request is legitimate.
* **Options**:
* `httpOnly`: The cookie is inaccessible to JavaScript, preventing attacks where an attacker could access the CSRF token.
* `sameSite: 'lax'`: Prevents the cookie from being sent with cross-origin requests, but allows it for same-site requests and top-level navigation.
* `secure`: Ensures the cookie is only sent over HTTPS to prevent interception.
### PKCE Code Verifier Cookie
* **Cookie Name**: `nile.pkce.code_verifier`
* **Purpose**: This cookie stores the code verifier used in the PKCE (Proof Key for Code Exchange) flow, which is commonly used in OAuth 2.0 authentication.
* **How it Works**: During OAuth authorization, the PKCE flow involves generating a code verifier and code challenge. The code verifier is stored in this cookie and used to validate the code challenge received from the OAuth provider, ensuring the request is secure and preventing token interception.
* **Options**:
* `httpOnly`: Prevents the cookie from being accessed via JavaScript.
* `sameSite: 'lax'`: Limits the cookie's availability to same-site requests, improving security.
* `secure`: Ensures the cookie is only sent over secure HTTPS connections.
* `maxAge: 900`: Sets the cookie's expiration time to 15 minutes (900 seconds), which is typical for OAuth PKCE flows.
### State Cookie
* **Cookie Name**: `nile.state`
* **Purpose**: This cookie stores the state parameter used to prevent CSRF attacks during the OAuth flow.
* **How it Works**: The state parameter is generated during the OAuth authorization request and stored in this cookie. It is later verified during the callback to ensure that the request originated from the legitimate application, protecting against CSRF attacks.
* **Options**:
* `httpOnly`: Prevents the cookie from being accessed by JavaScript, mitigating potential XSS attacks.
* `sameSite: 'lax'`: Limits cross-site cookie transmission to prevent potential cross-origin attacks.
* `secure`: Ensures the cookie is only sent over secure connections.
* `maxAge: 900`: Sets the cookie's expiration time to 15 minutes, aligning with the OAuth flow's expected duration.
### Nonce Cookie
* **Cookie Name**: `nile.nonce`
* **Purpose**: This cookie stores a nonce (a random value used once) that is used in the OAuth flow to protect against replay attacks.
* **How it Works**: The nonce is generated before redirecting the user to the OAuth provider and stored in this cookie. It is sent along with the OAuth request and verified by the server to ensure that the response is legitimate and has not been replayed by an attacker.
* **Options**:
* `httpOnly`: Ensures the cookie is not accessible to JavaScript, mitigating the risk of attacks that access browser storage.
* `sameSite: 'lax'`: Limits the cookie to same-site requests, preventing cross-site request forgery.
* `secure`: The cookie is only sent over HTTPS, providing security for sensitive data.
### Tenant cookie
* **Cookie Name**: `nile.tenant-id`
* **Purpose**: This cookie stores the tenant id, if selected. If it is present, the SDK and nile auth will use this value as the context for any query or request.
* **Options**:
* `sameSite: 'lax'`: Limits the cookie to same-site requests, preventing cross-site request forgery.
* `secure`: The cookie is only sent over HTTPS, providing security for sensitive data.
### Password reset cookie
* **Cookie Name**: `nile.reset`
* **Purpose**: This cookie stores a signed value that is allowed to reset a user's password.
* **Options**:
* `sameSite: 'lax'`: Limits the cookie to same-site requests, preventing cross-site request forgery.
* `secure`: The cookie is only sent over HTTPS, providing security for sensitive data.
## Related Topics
* [Sessions](/auth/concepts/sessions)
* [JWT](/auth/concepts/jwt)
* [Security](/auth/concepts/oauth)
# Extensions
Source: https://thenile.dev/docs/auth/concepts/extensions
Extend the SDK to automatically set context in your application
Extensions make it possible for `@niledatabase/server` package to integrate into developer applications easier.
It allows developers to hook into the life cycle of the SDK, from REST calls to db queries and set the context according to whatever method is being used to store user session information beyond cookies.
## Understanding context
`@niledatabase/server` contains three basic items that are controlled via context:
`headers`
`tenantId`
`userId`
Each of these can be set and will cause the SDK to use them either as a REST call or in a database query.
### Headers
Headers contain the auth cookies used to allows users access to `nile-auth`.
More often than not, a framework-specific extension will handle setting `headers` automatically when it comes to serving REST request to the client.
```ts nextjs theme={null}
// because NextJS uses an import, we must set it outside of the request lifecycle to be used later
export function nextJs = () => ({
id: 'next-js',
withContext:(ctx: CTX) => {
const { cookies, headers } = await import('next/headers');
const headersHelper = await headers();
const cooks = await cookies();
ctx.set({ headers: headersHelper });
// rest of context
},
// rest of extension
})
```
```ts express theme={null}
// express is one long callback chain, so execute from within the instance context directly.
export function express = (instance: Server) => ({
id: 'express',
onHandleRequest: async (
params: [ExpressRequest, ExpressResponse, NextFunction]
) => {
// ...
const proxyRequest = new Request(reqUrl, init);
const context = {
headers: new Headers(req.headers as HeadersInit),
tenantId: req.params?.tenantId || undefined,
};
const response = await instance.withContext(context, async (ctx) => {
return (await ctx.handlers[
method as 'GET' | 'POST' | 'PUT' | 'DELETE'
](proxyRequest, { disableExtensions: ['express'] })) as Response;
});
// ...
})
```
### tenantId and userId
The tenantId and userId are used to further authorize users in the database. There are many ways that a tenantId can be stored, but the most common ways would be stored in the URL or in a cookie.
Lets have a look at a cookie in express for illustrative purposes.
```ts naive approach theme={null}
app.get(async (req) => {
const tenantId = getTenant(req.headers); // your logic
const ctx = await nile.withContext({ tenantId });
// Use the contextualized instance
const users = await ctx.users.list();
});
```
Doing this manually on every route is repetitive and error-prone.
Instead, you can use Express middleware to call withContext once per request. The SDK caches the last used context, so any subsequent SDK calls will automatically use it within the same request chain:
```ts theme={null}
app.use((req, res, next) => {
const tenantId = getTenant(req.headers); // your logic
nile.withContext({ tenantId });
next();
});
app.get(async (req, res) => {
// Context is already set by the middleware
const users = await nile.users.list();
res.json(users);
});
```
This relies on synchronous execution within the same request lifecycle. If you
await or dispatch across async boundaries without using the SDK's
internal context propagation (like AsyncLocalStorage), the context may be
lost, which defaults to the last used.
Depending on where you are in the lifecycle of your application, you will want to call different functions within your extension.
## Extension methods
### withContext
Called when ever `withContext` is called, including internally by the SDK. Use this function set the `tenantId`, `userId` or `headers`
### onRequest
Called before the request is sent via `fetch` to `nile-auth`. the internal `Request` object and `ctx` methods are provided to the function.
### onResponse
Called immediately after the `fetch` completes. The raw `Response` object and `ctx` are provided to the function.
### onHandleRequest
Replaces the calls in the sdk that go to `nile-auth`. This function is provided the `params` from the calling framework. Useful for turning whatever framework's REST handlers into a standard `Request` object to be sent to `nile-auth` and then calling `nile-auth` directly from the extension.
Must return `ExtensionState.onHandleRequest` from the function to stop the standard handlers from being called.
### onConfigure
Allows for runtime configuration changes by extensions.
### withUserId
Called with `withContext`, allows overriding of a user id within the context.
### withTenantId
Called with `withContext`, allows overriding of a tenant id within the context.
### replace.handlers
Allows a full replacement of handlers that do not match any expected signature from the SDK. This is specifically for server side frameworks like Nitro, which expect a specific signature from `defineEventHandler` that the sdk does not have, as the standard signature is `const { GET, POST, PUT, DELETE } = nile.handlers`
# JWTs
Source: https://thenile.dev/docs/auth/concepts/jwt
Understanding JWT implementation in Nile Auth
## What is JWT?
At its core, JWT is just a JSON object that is efficiently encoded as a string.
However, encoded strings can be easily manipulated, so JWT also includes a signature that allows you to verify that the string has not been tampered with.
In the context of authentication, signed JWTs are used to represent authenticated user sessions.
When a user logs in, a signed JWT is created and sent to the user's browser.
The signed JWT includes the user's session information, such as the user's ID, email, and other properties - but most importantly,
it includes the server's cryptographic signature and an expiration time.
This JWT is then sent back to the server with subsequent requests, allowing the server to verify that the user is authenticated.
Because the JWT exists only on the client side, and is verified by the server, it has advantages and disadvantages:
* It can be used without cookies (for example, using the bearer authentication scheme for REST APIs)
* The server can verify the JWT without having to maintain a session on the server side or making database calls.
* It is easily manipulated, so it is important to verify the signature and check the expiration time.
* It cannot be revoked once issued, an issued JWT will remain valid until it expires.
## JWT Usage in Nile Auth
Nile Auth uses JWTs to represent user sessions for the email + password provider.
For other providers, database sessions are used instead.
You never need to deal with JWTs directly - Nile Auth and Nile SDK are responsible for creating, verifying and storing them.
You can access the current JWT using the `useSession` hook (client side), or by calling `nile.auth.getSession` (server side).
You can learn more about sessions in the [Sessions](/auth/concepts/sessions) documentation - which includes information about working with
user sessions whether they are represented by JWTs or database sessions.
However, it is useful to understand how JWTs work and what they contain.
### JWT Structure
The JWT used by Nile Auth contains the ID of the user (sub - the subject that the token refers to), the user's email, and the expiration time.
```json theme={null}
{
"sub": "1234567890",
"email": "test@example.com",
"exp": 1715027200
}
```
## Security Considerations
### Token Storage
Tokens should be stored securely on the client side - this typically means using secure, httpOnly cookies.
Storing tokens in local storage or session storage is not recommended, as this makes them susceptible to XSS attacks.
Nile Auth uses secure, httpOnly cookies by default. So you typically don't need to worry about this.
### Token Expiration
Nile Auth sessions expiration time is set to 30 days, and this applies to both database sessions and JWT sessions.
Because JWT sessions are stateless, they cannot be revoked once issued.
However, you can [remove the user from specific tenants](/auth/dashboard/managing-tenants) and disable their access to them.
## Best Practices
When working with JWTs, there are several important security practices to follow. Nile Auth implements all of these best practices by default:
* **Use Strong Signatures**: Always use strong cryptographic algorithms (like RS256) for signing JWTs. Never use weak algorithms or "none".
* **Keep Tokens Small**: JWTs should contain only essential claims to minimize token size and reduce overhead.
* **Secure Token Transmission**: Always transmit tokens over HTTPS and use secure, httpOnly cookies.
* **Validate All Claims**: Always verify the signature and validate all claims (especially expiration) before trusting a token.
* **Don't Store Sensitive Data**: Avoid storing sensitive information in JWTs as they are stored on the client side and are only signed, not encrypted.
## Related Topics
* [Sessions](/auth/concepts/sessions)
* [OAuth](/auth/concepts/oauth)
* [Cookies](/auth/concepts/cookies)
# Multi-factor Authentication (MFA) Concepts
Source: https://thenile.dev/docs/auth/concepts/multifactor
Understand the conceptual flow of Multi-factor Authentication (MFA) in Nile.
Multi-factor Authentication (MFA) adds an essential layer of security to user accounts by requiring users to provide two or more verification factors to gain access. This significantly reduces the risk of unauthorized access, even if a password is compromised.
Nile supports two primary MFA methods:
1. **Authenticator Apps**: Uses Time-based One-Time Password (TOTP) algorithms, where a code is generated by an authenticator app (e.g., Google Authenticator, Authy) on the user's device.
2. **Email One-Time Code (OTP)**: A temporary verification code is sent to the user's registered email address.
## The MFA Flow
The MFA lifecycle in Nile can be broken down into four key stages:
### 1. Setup (Enrollment)
During setup, a user registers an MFA method with their account. **Note: This step assumes the user is already signed up and currently logged in.**
* **Client-Side**: The client initiates the setup process.
* **React**: Use the [`useMultiFactor`](/auth/components/multifactor#usemultifactor-options) hook along with [` `](/auth/components/multifactor#authenticator-setup) (for apps) or [` `](/auth/components/multifactor#email-setup) (for email) to render the setup UI.
* **Client SDK**: Call the [`mfa`](/auth/sdk-reference/javascript/client/authorizer#mfa) function with `scope: 'setup'` directly from [`@niledatabase/client`](/auth/sdk-reference/javascript/client/authorizer).
* **Server-Side**: The `@niledatabase/server` package acts as a proxy, forwarding this request to the core Nile API.
* **API Response**: The Nile API responds with method-specific information:
* For **Authenticator**: An `otpauthUrl` (for QR code generation), a `secret`, and initial `recoveryKeys`.
* For **Email**: A `maskedEmail` and a `token` to be used for verification.
* **User Action**: The user typically scans a QR code (authenticator) or checks their email for a code.
```typescript theme={null}
import { auth } from '@niledatabase/client';
// Start enrollment for an authenticator app
const response = await auth.mfa({
method: 'authenticator',
scope: 'setup',
});
// Response contains secret, otpauthUrl (for QR), and recoveryKeys
// { method: 'authenticator', secret: '...', otpauthUrl: '...', ... }
```
### 2. Challenge
After an MFA method is set up, a "challenge" is issued whenever a user attempts to sign in (or perform certain sensitive actions). This requires the user to provide their second factor.
* **Sign-in Response**: When a user signs in with credentials, if MFA is enabled for their account, the sign-in response will often include a `token` and `method` indicating that an MFA challenge is required.
* **User Prompt**: The application prompts the user to enter a code.
* **React**: The [` `](/auth/components/multifactor#challenge-content-sign-in-or-disable) component is designed to handle this interaction seamlessly.
* **Custom UI**: Build a form to capture the code and submit it via the client SDK.
### 3. Verification
This is the core "verify" step where the user's provided code is checked against the registered MFA method.
* **Client-Side**: The client calls the `mfa` function with:
* `token`: The challenge token received from the sign-in response or setup.
* `code`: The 6-digit code or recovery code provided by the user.
* `scope`: Set to `'challenge'` to indicate verification.
* `method`: The specific MFA method being verified (`'authenticator'` or `'email'`).
* **Server-Side**: Again, the `@niledatabase/server` proxies this request.
* **API Response**:
* **Success**: Returns `{ ok: true, scope: 'challenge' }` (and potentially `recoveryCodesRemaining` if a recovery code was used).
* **Failure**: Returns an error, often coerced into a `{ url: string }` object (a `ChallengeRedirect`) with an `error` query parameter, which your application can then parse and display.
```typescript theme={null}
// Verify the code entered by the user
const verification = await auth.mfa({
token: 'token_from_setup_or_login',
code: '123456', // 6-digit code or recovery key
method: 'authenticator',
scope: 'challenge',
});
if (verification.ok) {
// Verification successful
}
```
### 4. Removal (Disable)
Users can remove an enrolled MFA method from their account.
* **Client-Side**: The client calls the `mfa` function with `remove: true` and the `method` to be disabled.
* **Server-Side**: Proxied through `@niledatabase/server`.
* **API Response**: On success, the MFA method is removed. In some cases, the API might issue a challenge before removal (e.g., to confirm the user's identity).
```typescript theme={null}
// Remove the authenticator method
await auth.mfa({
method: 'authenticator',
remove: true,
});
```
## User Object and MFA Status
The `User` object, retrieved via `auth.getSession()`, now includes a `multiFactor` property. This property indicates whether MFA is enabled for the user, allowing your application to dynamically adjust UI and workflows based on the user's MFA status.
```typescript theme={null}
const session = await auth.getSession();
if (session.user?.multiFactor) {
// User has MFA enabled
console.log('MFA Enabled');
}
```
## Configuration and Redirection
Nile Auth handles various redirections during the MFA flow. Specifically, when the backend requires a redirect (e.g., due to an error or successful completion of a step), it will return a `{ url: string }` object. This is referred to as a `ChallengeRedirect`. Your application should handle these redirects to guide the user through the appropriate next steps or display relevant messages based on the `error` query parameter in the URL.
## Related Resources
* [MFA Components](/auth/components/multifactor)
* [Client SDK Reference](/auth/sdk-reference/javascript/client/authorizer)
* [React Framework Guide](/auth/frameworks/react)
# OAuth
Source: https://thenile.dev/docs/auth/concepts/oauth
Understanding OAuth implementation in Nile Auth
Nile auth provides an easy-to-use, flexible OAuth implementation that allows your application to authenticate users via a variety of third-party OAuth providers (e.g., Google, Facebook, GitHub). This guide explains how to configure and use OAuth authentication in your database.
## OAuth Overview
OAuth is an open standard for authorization, which allows third-party services to securely access resources without exposing sensitive user credentials. NextAuth.js supports OAuth 2.0 and integrates with multiple authentication providers out of the box.
Nile auth allows you to configure OAuth providers, all while handling the authentication flows and securely storing and managing user sessions directly in your database.
## Setting Up OAuth Providers
### Authorization Code Flow
Your server acts as a proxy against Nile auth. A client interacts directly with your endpoints, which are forwarded on to Nile auth, which does all of the heavy lifting for you, all the while keeping it transparent to your users.
When a user signs in via an OAuth provider, the following flow occurs:
* The user clicks the login button for the desired provider (e.g., Google, GitHub).
* The user is redirected to the provider's login page.
* The user grants permission to your application.
* The provider redirects the user back to your application with an authorization code.
* Your server then exchanges the authorization code for an access token from nile auth, which in turn sends a payload back to give to the client.
* The user is authenticated, and a session is created.
## Provider Configuration
All providers can be configured on the **Configuration** screen under **Providers**, which is located in the **Tenant and Users** page
When using SSO providers and email + password, the `email_verified` column in
the `users.users` table must be set. An error will be produced if an SSO user
attempts to log in with email + password.
## Error Handling
Because your backend service proxy's Nile auth's API, you can intercept errors for your users and handle them accordingly (vs using the default pages).
In this example, a previously existing user has tried to log in with an email that already exists within the system, and it is tied to a different provider (eg the user used the same email in Google and Discord)
### Example error handling
`/api/auth/error/route.tsx`
```jsx theme={null}
import { redirect } from "next/navigation";
import { handlers } from "../../[...nile]/nile";
export async function GET(req: Request) {
const url = new URL(req.url);
if (url.searchParams.get("error") === "OAuthAccountNotLinked") {
redirect("/errors/oauth-not-linked");
}
return handlers.GET(req);
}
```
`/app/errors/oauth-not-linked.tsx`
```jsx theme={null}
import Link from 'next/link';
export default function OauthNotLinked() {
return (
Something went wrong.
You have selected a provider, but you have previously logged into the
app with a different one.
Go back and select the provider associated with this email
);
}
```
### OAuth Errors
If an error occurs in the OAuth flow, query params redirecting back to an error page will occur. Below are a list of errors:
**OAuthAccountNotLinked**
If the email on the account is already linked, but not with this OAuth account
**OAuthCallback**
Error in handling the response from an OAuth provider.
**OAuthCreateAccount**:
Could not create OAuth provider user in the database.
**EmailCreateAccount**
Could not create email provider user in the database.
**Callback**
Error in the OAuth callback
**EmailSignin**
Sending the e-mail with the verification token failed
There should also be further details logged when this occurs, such as the error is thrown, and the request body itself to aid in debugging.
## Related Topics
* [Single Sign On](/auth/singlesignon/google)
* [JWT](/auth/concepts/jwt)
* [Sessions](/auth/concepts/sessions)
# Per-Tenant Overrides
Source: https://thenile.dev/docs/auth/concepts/per-tenantoverrides
Understanding per-tenant configuration overrides in Nile Auth
Nile auth allows you to integrate various authentication providers (e.g., Google, Facebook, GitHub) into your application. You can configure and manage these providers globally and also override them for individual tenants to allow greater flexibility and control.
## Overview
Authentication providers are configured globally to provide users with the ability to sign in using third-party services. You can enable and configure multiple providers at once.
You can also customize which authentication providers are available for specific tenants. For example, you may want to disable the Google provider for one tenant, but keep it enabled for others.
## Configuration Areas
### OAuth Providers
To override (disable) a provider for a specific tenant
1. Navigate to the tenant in question **Tenants & Users**
2. Select your tenant from the table, then click **Configuration** from the sub menu.
3. De-select the tenant from the list of providers to disable it for that tenant.
## Related Topics
* [Tenants](/auth/concepts/tenants)
* [Built-in Tables](/auth/concepts/builtintables)
* [OAuth](/auth/concepts/oauth)
# Sessions
Source: https://thenile.dev/docs/auth/concepts/sessions
Understanding session management in Nile Auth
## What is a Session?
A session represents an authenticated user's state within your application. When a user successfully logs in, a session is created, storing information like the user's ID, email, and any custom properties you choose. This session allows the user to remain authenticated across multiple requests without needing to re-enter credentials.
Nile auth has two kinds of session tokens: JWT and database session tokens. For email + password, JWTs are used.
For all other providers, database session tokens are used.
## Accessing sessions
You can access the session client side by using `useSession`. This is rare, however, as the session contains the bare minimum information required for authorization and authentication. It is more likely you will use API requests to return information about the user (for instance, `useMe()` to get user profile information)
```jsx theme={null}
import { useSession } from '@niledatabase/react';
function Profile() {
const { data: session } = useSession();
if (!session) {
return You are not logged in.
;
}
return You are logged in
;
}
export default Profile;
```
You can access the session server side by using `nile.auth.getSession`.
```jsx theme={null}
import { Nile } from '@niledatabase/server';
const nile = Nile();
app.get('/some-path', async (req, res) => {
const session = await nile.auth.getSession(req);
if (!session) {
res.status(401).json({
message: 'Unauthorized',
});
return;
}
res.json({
message: 'You are authorized',
});
});
```
## Session Expiry
The default expiry time is 30 days. When a session expires, the user will need to log in again to create a new session.
## Revoking sessions
You can revoke a database session by deleting it from the database. This will cause the user to be logged out of *all* tenants the next time they make a request.
JWT sessions exist on the client side, so they cannot be revoked.
## Related Topics
* [JWT](/auth/concepts/jwt)
* [Users](/auth/concepts/users)
* [Cookies](/auth/concepts/cookies)
# Tenants
Source: https://thenile.dev/docs/auth/concepts/tenants
Learn about tenant management in Nile Auth
This guide explains the concept of tenants in Nile Auth and how to manage them effectively within your application.
## What is a Tenant?
A **tenant** in Nile Auth represents a logical partition of the system, allowing for isolated user data and configurations. Each tenant operates independently with its own set of users, resources, and configurations, while sharing common underlying infrastructure. This is a fundamental feature for multi-tenant applications.
## Tenant Features
* **Multi-tenancy support**: Handle multiple tenants within the same application without interference.
* **Tenant isolation**: Each tenant's data is isolated, ensuring privacy and security.
* **Tenant-specific configurations**: Customize settings, resources, and behavior per tenant.
## Working with Tenants
To manage tenants, you can interact with Nile Auth's API and middleware. Below is an example of how you can set the `tenantId` based on the URL parameter and handle tenant-specific tasks:
```typescript theme={null}
// Middleware to set the tenantId from the URL parameter
app.param('tenantId', (req, res, next, tenantId) => {
nile.withContext({ tenantId });
next();
});
// get all tasks for tenant
app.get('/api/tenants/:tenantId/todos', async (req, res) => {
// No need for a "where" clause here because we are setting the tenant ID in the context
const todos = await nile.query(
`SELECT * FROM todos
ORDER BY title`,
);
res.json(todos.rows);
});
```
It is also possible to set a tenant id in a header using `niledb-tenant-id`
## Best Practices
* **Ensure proper tenant isolation**: Use tenant-specific identifiers (e.g., `tenantId`) in all queries and resource access to prevent cross-tenant data leakage.
* **Handle authentication per tenant**: Authenticate users and sessions per tenant to maintain secure access controls.
* **Configure tenant-specific limits**: For example, limit the number of tasks per tenant or customize resource allocations.
## Related Concepts
* [Users](/auth/concepts/users)
* [Sessions](/auth/concepts/sessions)
* [JWT](/auth/concepts/jwt)
***
# Users
Source: https://thenile.dev/docs/auth/concepts/users
Understanding user management in Nile Auth
Learn about user management concepts in Nile Auth, including how to create, update, and manage users and their sessions.
## User Model
The `User` model in Nile Auth defines the structure of a user object. It includes both basic and custom properties to meet the needs of your application.
```typescript theme={null}
interface User {
id: string;
email: string;
name?: string | null;
familyName?: string | null;
givenName?: string | null;
picture?: string | null;
created: string;
updated?: string;
emailVerified?: string | null;
tenants: { id: string }[];
}
```
## User Properties
### Basic Properties
* **Email**: The user's email address. This is true across all users of a database
* **Profile information**: Includes details like the user's name, contact info, and other relevant data, along with any tenants the user is associated.
## User Operations
### Creating Users
To create a new user, you can make a POST request to the `/users` endpoint.
```typescript theme={null}
const newUser = await nile.users.createUser({
email: 'user@example.com',
password: 'user1',
});
```
### Adding users
When a user is created, they are not automatically added to a tenant, unless `newTenant` is present. Users can only add other users to tenants of which they are a member. To add a user to a tenant, use `linkUser`
```typescript theme={null}
const user1 = await nile.users.createUser({
email: 'user1@example.com',
password: 'user1',
newTenant: 'myTenant',
});
const user2 = await nile.users.createUser({
email: 'user2@example.com',
password: 'user2',
});
// make user1 and user2 part of the same tenant
const updated2 = await nile.tenants.addMember(user2.id);
```
### Updating Users
Because users are isolated to their own session, existing user update themselves via `PUT` method. A custom endpoint would need to be created in order for one user to update the information of another.
```typescript theme={null}
const updatedUser = await nile.users.updateSelf({ name: 'user1' });
```
### Deleting Users
Because user accounts are isolated, one user is unable to delete another. It is possible remove a user from a tenant from the built-in API. In order to to that, use `unlinkUser`
```typescript theme={null}
const unlinked = await nile.tenants.removeMember(user2.id);
```
## User Authentication
### Password-based Authentication
Password-based authentication is not recommended for production applications.
Use social authentication or other secure methods instead.
You can authenticate users using their email and password. After a successful login, a session token in for form of a JWT is returned. All other forms use a session token saved in the database.
For demonstration purposes, we are using the server-side methods. It would be rare to do this in a real application.
```typescript theme={null}
// Example password authentication - not recommended for production
const session = await nile.auth.login({
email: 'user@example.com',
password: 'password123',
});
```
### Social Authentication
Nile Auth also supports social authentication via OAuth providers such as Google or Facebook. In order to configure this, see [Single Sign On](/auth/singlesignon/google)
## User Sessions
A session is always within the context of a request. You can access session data using:
```typescript theme={null}
const session = await nile.auth.getSession();
```
## Related Topics
* [Sessions](/auth/concepts/sessions)
* [Tenants](/auth/concepts/tenants)
* [JWT](/auth/concepts/jwt)
# Development
Source: https://thenile.dev/docs/auth/contributing/develop
Learn how to contribute to Nile Auth development
## Getting Started
### Prerequisites
* Node.js (v16 or higher)
* npm or yarn
* Git
* PostgreSQL (for local development)
### Setting Up the Development Environment
1. Clone the repository:
```bash theme={null}
git clone https://github.com/niledatabase/niledatabase
cd niledatabase
```
2. Install dependencies:
```bash theme={null}
npm install
```
3. Set up environment variables:
```bash theme={null}
cp .env.example .env
```
## Project Structure
```
src/
├── auth/ # Authentication core
├── components/ # React components
├── hooks/ # Custom hooks
├── utils/ # Utility functions
└── types/ # TypeScript types
```
## Development Workflow
### Running Locally
1. Start the development server:
```bash theme={null}
npm run dev
```
2. Run tests:
```bash theme={null}
npm test
```
### Code Style
We use the following tools for code quality:
* ESLint for linting
* Prettier for code formatting
* TypeScript for type checking
### Making Changes
1. Create a new branch:
```bash theme={null}
git checkout -b feature/your-feature-name
```
2. Make your changes and commit:
```bash theme={null}
git add .
git commit -m "feat: description of your changes"
```
3. Push your changes:
```bash theme={null}
git push origin feature/your-feature-name
```
## Pull Request Process
1. Update documentation for any new features
2. Ensure all tests pass
3. Update the changelog
4. Submit your PR with a clear description
## Documentation
### Adding New Documentation
1. Create new MDX files in the appropriate directory
2. Update the navigation in `mint.json`
3. Include code examples and explanations
### Documentation Style Guide
* Use clear, concise language
* Include code examples
* Add proper headings and sections
* Link to related topics
## Best Practices
* Write clean, maintainable code
* Add unit tests for new features
* Keep components small and focused
* Document your code thoroughly
## Getting Help
* [Join our Discord](https://discord.com/invite/8UuBB84tTy)
* [Report Issues](/auth/contributing/report)
* [Read Testing Guide](/auth/contributing/testing)
## Related Topics
* [Testing Guide](/auth/contributing/testing)
* [Issue Reporting](/auth/contributing/report)
* [Community](/auth/help/community)
# Issues
Source: https://thenile.dev/docs/auth/contributing/report
Learn how to report issues and contribute to Nile Auth improvement
## Before Reporting
### Check Existing Issues
1. Search the [GitHub Issues](https://github.com/niledatabase/niledatabase/issues) and [GitHub Discussions](https://github.com/orgs/niledatabase/discussions)
2. Review closed issues
3. Check the documentation
### Gather Information
* Nile Auth version
* Environment details
* Steps to reproduce
* Error messages
* Relevant logs
## Creating an Issue
### Issue Template
```markdown theme={null}
### Description
[Clear description of the issue]
### Environment
- Nile Auth Version:
- Node.js Version:
- Operating System:
- Browser (if applicable):
### Steps to Reproduce
1. [First Step]
2. [Second Step]
3. [and so on...]
### Expected Behavior
[What you expected to happen]
### Actual Behavior
[What actually happened]
### Additional Context
[Any other relevant information]
```
## Issue Types
### Bug Reports
* Describe the bug clearly
* Provide reproduction steps
* Include error messages
* Add relevant code snippets
### Feature Requests
* Explain the use case
* Describe expected behavior
* Provide examples
* Consider alternatives
### Documentation Issues
* Identify unclear sections
* Suggest improvements
* Point out errors
* Propose additions
## Best Practices
### Writing Good Issues
1. Be specific and clear
2. One issue per report
3. Include relevant details
4. Use proper formatting
### Issue Labels
* bug: Bug reports
* enhancement: Feature requests
* documentation: Doc updates
* help wanted: Community assistance needed
* auth: Mark Nile Auth specific issues (not issues with the database, for instance)
## Contributing Solutions
### Fixing Issues
1. Comment on the issue
2. Fork the repository
3. Create a fix
4. Submit a pull request
### Pull Request Guidelines
```markdown theme={null}
### Related Issue
Fixes #[issue number]
### Changes Made
- [Change 1]
- [Change 2]
- [Change 3]
### Testing Done
- [Test 1]
- [Test 2]
### Screenshots (if applicable)
[Add screenshots]
```
## Security Issues
### Responsible Disclosure
1. **DO NOT** create public issues
2. Email [security@thenile.dev](mailto:security@thenile.dev)
3. Wait for response
4. Follow disclosure timeline
### Security Guidelines
* Report vulnerabilities privately
* Provide proof of concept
* Allow time for fixes
* Follow up responsibly
## Getting Help
* [Join Discord Community](https://discord.com/invite/8UuBB84tTy)
* [Read Documentation](/auth/introduction)
* [Contact Support](https://www.thenile.dev/contact-us)
## Issue Lifecycle
1. Submission
2. Triage
3. Investigation
4. Resolution
5. Closure
## Best Practices
* Be respectful and professional
* Follow up on your issues
* Help others when possible
* Keep discussions focused
## Related Topics
* [Development Guide](/auth/contributing/develop)
* [Testing Guide](/auth/contributing/testing)
* [Community Guidelines](/auth/help/community)
# Testing
Source: https://thenile.dev/docs/auth/contributing/testing
Learn how to test your contributions to Nile Auth
## Overview
### Types of Tests
* Unit Tests
* Integration Tests
* End-to-End Tests
* Performance Tests
## Unit Testing
### Writing Unit Tests
```typescript theme={null}
// Example unit test
import { describe, it, expect } from 'vitest';
import { validateEmail } from '../utils/validation';
describe('validateEmail', () => {
it('should validate correct email format', () => {
expect(validateEmail('test@example.com')).toBe(true);
});
});
```
### Test Coverage
* Aim for 80% or higher coverage
* Focus on critical paths
* Test edge cases
## Integration Testing
### Setting Up Integration Tests
```typescript theme={null}
// Example integration test
import { describe, it, expect } from 'vitest';
import { AuthClient } from '../client';
describe('AuthClient', () => {
it('should authenticate user', async () => {
// Placeholder test implementation
});
});
```
### Test Environment
1. Configure test database
2. Mock external services
3. Set up test fixtures
## End-to-End Testing
### E2E Test Setup
```typescript theme={null}
// Example E2E test
import { test, expect } from '@playwright/test';
test('user can sign in', async ({ page }) => {
// Placeholder E2E test
});
```
### Running E2E Tests
1. Start test environment
2. Execute test suite
3. Generate reports
## Performance Testing
### Load Testing
```typescript theme={null}
// Example load test
import { check } from 'k6';
import http from 'k6/http';
export default function () {
// Placeholder load test
}
```
### Benchmarking
* Response times
* Concurrent users
* Resource usage
## Test Best Practices
### Writing Tests
* Keep tests focused and simple
* Use descriptive test names
* Follow AAA pattern (Arrange, Act, Assert)
* Mock external dependencies
### Test Organization
```
tests/
├── unit/
├── integration/
├── e2e/
└── performance/
```
## Continuous Integration
### CI Pipeline
1. Run linting
2. Execute unit tests
3. Run integration tests
4. Perform E2E tests
5. Check coverage
### Test Automation
```yaml theme={null}
# Example GitHub Actions workflow
name: Tests
on: [push, pull_request]
jobs:
test:
runs-on: ubuntu-latest
steps:
# Placeholder CI steps
```
## Debugging Tests
### Common Issues
* Async timing problems
* Test isolation
* Environment setup
* Flaky tests
### Tools and Techniques
* Test debugger
* Logging
* Snapshot testing
* Test runners
## Best Practices
* Write tests before code (TDD)
* Keep tests maintainable
* Use test fixtures
* Document test requirements
## Related Topics
* [Development Guide](/auth/contributing/develop)
* [Issue Reporting](/auth/contributing/report)
* [Contributing Overview](/auth/contributing/develop)
# Configurations
Source: https://thenile.dev/docs/auth/dashboard/configurations
Configure Nile Auth settings in the Dashboard
Nile Auth works out of the box with minimal configuration.
In this guide, we'll walk you through the various configurations available and how to use them, both in [Nile Console](https://console.thenile.dev) and in your application.
## Authentication Settings
### Sign-in Methods
Nile auth supports a variety of authentication methods. To manage them, navigate to the "Tenants and Users" page in the
Dashboard and select the "Configure" tab.
### Email Configuration
If you chose to use email as the authentication method, you'll likely want to set up email verification.
To verify users' emails, you'll need a way to send them a verification email and customize
your email templates. Nile provides full control over your email templates, allowing you to
maintain consistent branding and messaging.
You'll find both the SMTP settings and the email templates in the "Configure" tab, under "Email templates". As always in Nile, the templates, the variables and the SMTP settings are stored in *your* database, in the `auth` schema.
You can learn more about email verification in the [Email Verification](/auth/email/verification) guide.
### SMTP Settings
Nile Auth supports email providers like SendGrid, Mailgun, and others.
You'll need to sign up for an account with one of these providers and then enter the SMTP URI, including credentials.
In addition, you'll need to set the "Sender" email address to a valid email address in your domain.
As you can see, in addition to the "Sender" email address, you can provide other variables.
These will be used in the email templates.
You can learn more about SMTP settings in the [Custom SMTP](/auth/email/customsmtp) guide.
### Email Templates
Once you've set up the SMTP settings, you can customize the email templates.
You'll find two templates in the "Email templates" section:
* **Email invitation**: Sent when a user uses their email address to sign up (whether with magic link, through an invitation or with a password).
* **Reset password**: Sent when a user requests a password reset.
Click on the template you want to customize and you'll see the template editor. Email templates are just HTML, so you can customize them as you like.
You'll find the template variables that are available on the left side of the template editor, for convenience.
You can learn more about email templates in the [Email Templates](/auth/email/templates) guide.
## Social Providers
You can configure social providers in the "Providers" tab.
Simply click on the provider you want to configure and you'll see the provider configuration page.
All providers use OAuth, so you'll need to provide the OAuth client ID and secret,
which you will get from the provider when you configure it in their developer console.
You can see more details on how to configure and use each provider in the [Single Sign-On](/auth/singlesignon/google) section of the documentation.
## Per-Tenant Configuration
In addition to setting up authentication for your application as a whole,
you can also configure authentication providers on a per-tenant basis.
To do this, navigate to the "Tenants and Users" page in Nile console, select the tenant
you want to configure and click on the "Configuration" tab.
## Application Configuration
When you use Nile SDK with your application, you'll need to tell the application where to find the database and the auth service.
### Environment Variables
You'll find all the environment variables you need in the [Nile Console](https://console.thenile.dev) under the "Settings" tab.
Make sure you generate credentials, and then copy the configuration to your `.env` file:
```bash theme={null}
NILEDB_USER=niledb_user
NILEDB_PASSWORD=niledb_password
NILEDB_API_URL=https://us-west-2.api.thenile.dev/v2/databases/
NILEDB_POSTGRES_URL=postgres://us-west-2.db.thenile.dev:5432/
```
### Instantiate the SDK
When using the SDK, you'll need to instantiate it:
```typescript theme={null}
const _nile = await NileServer({});
```
Typically, you don't need to pass any configuration, since the SDK will automatically use the environment variables you set in the `.env` file.
However, if you need to, you can pass a configuration object to the `NileServer` constructor. Here's the type definition for the configuration object and the matching environment variables. The debug flag is useful to enable debug logging while developing.
```typescript theme={null}
export type ServerConfig = {
databaseId?: string; // process.env.NILEDB_ID
user?: string; // process.env.NILEDB_USER
password?: string; // process.env.NILEDB_PASSWORD
databaseName?: string; // process.env.NILEDB_NAME
tenantId?: string | null | undefined; // process.env.NILEDB_TENANT
userId?: string | null | undefined;
debug?: boolean;
configureUrl?: string; // process.env.NILEDB_CONFIGURE
secureCookies?: boolean; // process.env.NILEDB_SECURECOOKIES
db?: NilePoolConfig; // db.host process.env.NILEDB_HOST
api?: {
version?: number;
basePath?: string; // process.env.NILEDB_API
cookieKey?: string;
token?: string; // process.env.NILEDB_TOKEN
};
logger?: LoggerType;
};
```
## Best Practices
* Always configure SMTP settings with a production-grade email provider to ensure reliable delivery of verification and password reset emails
* Enable only the authentication methods you plan to actively maintain and support.
* Some authentication providers (Notably Google) require approval process, so it is recommended to start setting them up in advance.
* Store environment variables securely and never commit them to version control.
* For multi-tenant applications, review and test authentication settings for key tenants individually.
* Customize email templates to match your brand and provide clear instructions to users.
* Enable debug logging only in development environments - it is not recommended to use it in production for both performance and security reasons.
## Related Topics
* [Managing Users](/auth/dashboard/managing-users)
* [Managing Tenants](/auth/dashboard/managing-tenants)
* [Email Templates](/auth/email/templates)
# Managing Tenants
Source: https://thenile.dev/docs/auth/dashboard/managing-tenants
Learn how to manage tenants using the Nile Auth Dashboard
Nile Console includes a dashboard for managing tenants, as part of Nile's multi-tenant authentication. It allows you, as a developer, to create, edit, and delete tenants,
as well as manage users and configurations for each tenant.
This guide explains how to manage tenants using [Nile Console](https://console.thenile.dev).
The same functionality is also available in the [Nile CLI](/cli/introduction).
## Dashboard Overview
You can access the tenant management dashboard by navigating to the "Tenants and Users" tab in the left sidebar of Nile Console.
## Tenant Management Features
* Creating new tenants
* Editing tenant settings
* Managing tenant users
* Configuring tenant-specific settings
## Common Tasks
### Creating a New Tenant
To create a new tenant, click the "Create Tenant" button in the top right corner of the dashboard.
Then fill in the tenant name and click "Create".
This is the equivalent of running `insert into tenants (name) values ('')` in SQL or `nile tenants create --name ` in the CLI.
### Configuring Tenant Settings
To configure tenant settings, click on the tenant you want to configure.
In the "Profile" tab, you can modify the tenant's name. Simply click on the pencil icon next to the current name and
type in the new name.
Nile Auth also supports tenant-specific authentication providers.
To configure a tenant's authentication provider, click on the "Configuration" tab and select the providers you want to use.
### Managing Tenant Users
To manage tenant users, click on the tenant you want to manage and then click the "Users" tab.
In this tab, you can see a list of all the users that are part of the tenant.
To remove a user from the tenant, click the "Delete" button next to the user you want to delete.
Note that this will revoke the user's access to the tenant, but they will still exist in the database
and can still authenticate with the other tenants.
You can also edit the user's name or email address by clicking on the field and typing in the new value.
You can click the "Add User" button to add an existing user to the tenant.
Or you can create new users and they will be added to the tenant automatically:
### Deleting a Tenant
To delete a tenant, click on the tenant you want to delete and then click the "Delete" button.
Note that the tenant will not be deleted from the database - this is a soft delete, it will be marked as deleted and users will
no longer be able to access it. But the tenant and all of its data will still exist in the database and can be undeleted if needed.
If you need to truly delete the tenant, you can do this directly in the database by running the following SQL command:
```sql theme={null}
DELETE FROM tenants WHERE id = '';
```
This guide explains how to manage tenants using the Nile Auth Dashboard.
## Best Practices
* Give tenants clear, descriptive names that help identify the organization (e.g., "Acme Corp - Production" rather than just "Acme")
* The tenants table is yours! You can add fields to it to store any information you need about the tenant - stripe customer id, subscription tier, etc.
* Remove inactive users promptly to maintain security
* Regularly audit tenant access permissions
* Monitor for unusual activity patterns
## Related Topics
* [Managing Users](/auth/dashboard/managing-users)
* [Configurations](/auth/dashboard/configurations)
# Managing Users
Source: https://thenile.dev/docs/auth/dashboard/managing-users
Learn how to manage users using the Nile Auth Dashboard
Nile console includes a dashboard for managing users, as part of Nile's multi-tenant authentication. It allows you, as a developer, to create, edit, and delete users.
This guide explains how to manage users using [Nile Console](https://console.thenile.dev).
## Dashboard Overview
You can access the user management dashboard by navigating to the "Tenants and Users" page in the left sidebar of Nile Console and then selecting the "Users" tab.
Since the number of users is typically large, the dashboard has a search bar to help you find the users you need.
### User Details View
You can click on a user to view their details:
## User Management
### Creating Users
You can create a user by clicking on the "Create User" button in the top right corner of the dashboard.
Filling in the required fields and clicking on the "Create" button will create a new user.
Creating a user in this dashboard will not add them to any tenant (you can do
that later in the [tenants dashboard](/auth/dashboard/managing-tenants)) and
will not send them an email to verify their account.
### Editing Users
You can edit a user by clicking on the field you want to change and starting to type.
This works both in the top level list view and in the user details view.
### Deleting Users
You can delete a user by clicking on the "Delete" button in the user list view.
## User Properties
All user properties are stored in the `users.users` table. The dashboard, APIs
and SDK all use this table and therefore have access to the same properties.
```bash theme={null}
Table "users.users"
Column | Type | Collation | Nullable | Default
----------------+-----------------------------+-----------+----------+---------------------------
id | uuid | | not null | public.uuid_generate_v7()
created | timestamp without time zone | | not null | LOCALTIMESTAMP
updated | timestamp without time zone | | not null | LOCALTIMESTAMP
deleted | timestamp without time zone | | |
name | text | | |
family_name | text | | |
given_name | text | | |
email | text | | |
picture | text | | |
email_verified | timestamp without time zone | | |
Indexes:
"users_pkey" PRIMARY KEY, btree (id)
"users_email_key" UNIQUE, btree (email) WHERE deleted IS NULL
Referenced by:
TABLE "auth.credentials" CONSTRAINT "credentials_user_id_fkey" FOREIGN KEY (user_id) REFERENCES users(id)
TABLE "auth.oidc_auth_attempts" CONSTRAINT "oidc_auth_attempts_user_id_fkey" FOREIGN KEY (user_id) REFERENCES users(id)
TABLE "auth.sessions" CONSTRAINT "sessions_userId_fkey" FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE
```
This single source of truth makes it easier to manage user data, compared to auth systems that store user information in their own data stores.
The SDK uses the following interface to represent a user:
```typescript theme={null}
export interface User {
id: string;
email: string;
name?: string | null;
familyName?: string | null;
givenName?: string | null;
picture?: string | null;
created: string;
updated?: string;
emailVerified?: string | null;
tenants: { id: string }[];
}
```
## User Account Verification
It is common to want to verify a user's email address before allowing them to access your application.
This is especially important for email/password users since there is no other way to know if the email address is valid.
Nile Auth APIs and SDK support email verification with the ` ` component.
You can learn how to use it in the [Email Verification](/auth/email/verification) guide.
In addition, you may want to verify (or unverify) a user's email address manually.
You can do this by updating the `emailVerified` property in the `users.users` table.
This will mark the user as verified starting from the current time.
```sql theme={null}
UPDATE users.users SET email_verified = CURRENT_TIMESTAMP WHERE id = '';
```
To unverify a user, you can set the `emailVerified` property to `NULL`.
```sql theme={null}
UPDATE users.users SET email_verified = NULL WHERE id = '';
```
## Best Practices
* Always use meaningful names when creating users to make them easily identifiable in the dashboard
* Don't modify existing user properties without approval from the user or an administrator for their tenant.
* Regularly review user accounts and remove inactive or invalid ones to maintain security. Nile's users dashboard has a helpful "Last Login" column that can help you identify inactive users.
* Verify email addresses to ensure that they are valid and to prevent spam.
* Use the search functionality to quickly find users instead of scrolling through the list.
* Document any manual changes made to user accounts for audit purposes
## Related Topics
* [Managing Tenants](/auth/dashboard/managing-tenants)
* [Configurations](/auth/dashboard/configurations)
* [User Management Concepts](/auth/concepts/users)
# Custom SMTP
Source: https://thenile.dev/docs/auth/email/customsmtp
Configure custom SMTP servers in Nile Auth
1. Navigate to **Tenants & Users -> Configuration -> Email templates**.
2. Enter the name and SMTP settings for your custom server. The format should be `smtp://username:password@emailservice:587`, which would be provided by the email service
3. Enter an `app_name` and `sender`. These values will be used by the Nile auth to send your emails.
Be sure that **`sender`** is a valid email address. Some email services will
reject emails that do not have a valid sender address.
Once your SMTP server is configured, you can test the connection to ensure
that it is working correctly. Select a template and at the bottom, click the
**Test email sending** button. It will send an email to the email address of
your developer account.
Most services require you to configure a test email. Use the email address
associated with the single sign on account you used when you logged in.
You can use variables in your email templates to personalize the content. The following variables are available automatically:
* **user.email** - the email of the user
* **user.name** - if set, the name of the user
* **api\_url** - the URL of your application. This is used to generate links in the email that come back to the server handling Nile auth requests. By default, this is obtained from the headers of the request, but it is possible to hardcode and/or overide it
Additional variables can be used and will be replaced in the HTML sent using the `${variable}` syntax. For example
```html theme={null}
```
* [Email Templates](/auth/email/templates)
* [Email Verification](/auth/email/verification)
* [Per-tenant Overrides](/auth/concepts/per-tenantoverrides)
# null
Source: https://thenile.dev/docs/auth/email/invites
## Overview
Email invites allow users to invite additional users to their tenants.
When a user invites another user, Nile Auth:
1. Sends an invite email using your configured SMTP server.
2. Grants access to the user after they click the invite link.
## Prerequisites
* A configured SMTP server is required to send verification emails. You can set this up in the Nile Auth UI under **Tenants & Users -> Configuration -> Email templates**.
* A web server configured to handle Nile Auth requests and serve html.
## Implementation
```bash theme={null}
npm install @niledatabase/server @niledatabase/react
@niledatabase/client @niledatabase/nextjs
```
Your application must expose API routes to handle authentication operations.
Create a folder called `api` under the `app` folder and a folder called `[...nile]` under it:
```bash theme={null}
mkdir -p app/api/\[...nile\]
```
Create following files handle the calls to your server, as well as expose the `nile` instance to your application:
```typescript app/api/[...nile]/nile.ts theme={null}
import { Nile } from '@niledatabase/server';
import { nextJs } from '@niledatabase/nextjs';
export const nile = Nile({ extensions: [nextJs] });
```
```typescript app/api/[...nile]/route.ts theme={null}
import { nile } from './nile';
export const { POST, GET, DELETE, PUT } = nile.handlers;
```
```typescript theme={null}
import { nile } from "@/lib/nile";
import { DataTable } from '@/components/table'
export function Invites() {
const [invites, users] = nile.withContext(
// better to save tenantId with a cookie or get with extension, but this works too
{ tenantId: parseTenantId(await headers()) },
async(_nile) => Promise.all([
_nile.tenants.invites();
_nile.tenants.users()
]);
);
return (
{/*A table for pending invites*/}
{/*A table for of existing users*/}
);
}
```
Based on the current context, invite the new user
```ts theme={null}
export async function inviteUser(
_: unknown,
formData: FormData
): Promise {
'use server';
const email = formData.get('email') as string;
}); // tenant context set by extension, else you need `nile.withContext` here
const response = await nile.tenants.invite({
email,
callbackUrl: `/your-callback-handler`
})
if (response instanceof Response) {
return {
ok: false,
message: `Failed to create invite for user: ${await response.text()}`,
};
}
return { ok: true, data: response };
}
```
For completeness, it is possible that an existing user invited a brand new user. Because user authorization/authentication is separate from tenant membership, and even invites exist outside of a user account, you may need to prompt the user to create an account that they are able to use.
In the below sample code, the `/your-callback-handler` could check if the user has logged in by virtue of the `nile.users.getSelf()` function.
If they are not logged in, send them to a sign up page for user creation or sign in.
If they are logged in, we check to be sure they are the same user. If not, they need to switch users (their current signed in user does not have access to the tenant, after all)
Lastly, if they are signed in as the user that was invited, they can see a list of invites on that tenant.
Nile-auth email address are an exact match. Some email providers (like gmail) will allow receiving emails that don't exactly match, eg ([my.cool.email@gmail.com](mailto:my.cool.email@gmail.com) can receive mail from [mycoolemail@gmail.com](mailto:mycoolemail@gmail.com)). In nile-auth, those are considered two separate users.
```ts theme={null}
import { User } from '@niledatabase/server';
import { redirect } from 'next/navigation';
import { NextRequest } from 'next/server';
import { nile } from '@/app/api/[...nile]/nile';
export async function GET(req: NextRequest) {
// you may have already been logged in, so we need to check
const me = await nile.users.getSelf();
if (me instanceof Response) {
// its a 404/401, which means the user needs to sign up before they can do any thing
return redirect('/invites/sign-up');
}
// we need to be sure the identifier matches the user. If not, we need to give them the option to switch users.
const email = req.nextUrl.searchParams.get('email');
if (email !== me.email) {
return redirect('/invites/user-switcher');
}
return redirect('/invites');
}
```
## Related Topics
* [Email Templates](/auth/email/templates)
* [Email Verification](/auth/email/verification)
* [Custom SMTP](/auth/email/customsmtp)
# Passwords
Source: https://thenile.dev/docs/auth/email/password
Learn about password management features in Nile Auth
## Features
* Password reset
* Password policies
* Password hashing and security
## Overview
With an SMTP provider configured on your database, users with emails can reset their passwords with a two step process. First, the users requests a link to be sent to their email address. The auth service creates a token (saved to the `auth.verification_tokens` table in your database), and sends it in an email. The token is valid for 4 hours by default.
## Installation
```sh theme={null}
npm install @niledatabase/react @niledatabase/client
```
## Usage
`/user/reset-password`
```tsx theme={null}
import { PasswordResetRequestForm } from './path/to/ResetForm';
const MyComponent = () => {
return ;
};
```
Once the user clicks on the link, they will be redirected to your app and will be able to securely reset their password.
`/user/change-password`
```tsx theme={null}
console.log('Password updated successfully')}
onError={(error) => console.error('Update failed', error)}
/>
```
## API
### `PasswordResetRequestForm` Props
| Name | Type | Default | Description |
| --------------- | ------------------------------------------------------- | -------------------------------------------------- | --------------------------------------------------------- |
| `client` | `QueryClient` *(optional)* | `undefined` | React Query client instance. |
| `callbackUrl` | `string` *(optional)* | `undefined` | The URL to redirect to after a successful password reset. |
| `defaultValues` | `object` *(optional)* | `{ email: '', password: '', confirmPassword: '' }` | Default form values. |
| `beforeMutate` | `(data: AllowedAny) => AllowedAny` *(optional)* | `undefined` | Function executed before mutation to modify request data. |
| `onSuccess` | `(res: Response) => void` *(optional)* | `undefined` | Callback triggered on successful password reset. |
| `onError` | `(error: Error, data: AllowedAny) => void` *(optional)* | `undefined` | Callback triggered if the reset process fails. |
### `PasswordResetForm` Props
| Name | Type | Default | Description |
| --------------- | ------------------------------------------------------- | -------------------------------------------------- | --------------------------------------------------------- |
| `client` | `QueryClient` *(optional)* | `undefined` | React Query client instance. |
| `callbackUrl` | `string` *(optional)* | `undefined` | The URL to redirect to after a successful password reset. |
| `defaultValues` | `object` *(optional)* | `{ email: '', password: '', confirmPassword: '' }` | Default form values. |
| `beforeMutate` | `(data: AllowedAny) => AllowedAny` *(optional)* | `undefined` | Function executed before mutation to modify request data. |
| `onSuccess` | `(res: Response) => void` *(optional)* | `undefined` | Callback triggered on successful password reset. |
| `onError` | `(error: Error, data: AllowedAny) => void` *(optional)* | `undefined` | Callback triggered if the reset process fails. |
## Internal Functionality
* Uses `useForm` from React Hook Form to handle form state.
* Calls `useResetPassword` with the provided email and password upon form submission.
* Implements password confirmation validation to ensure the user inputs matching passwords.
* Supports lifecycle hooks (`beforeMutate`, `onSuccess`, `onError`) for request customization.
* Disables the submit button if passwords do not match or required fields are missing.
## Related Topics
* [Email Templates](/auth/email/templates)
* [Email Verification](/auth/email/verification)
# Templates
Source: https://thenile.dev/docs/auth/email/templates
Customize email templates in Nile Auth
Nile Auth provides a default set of email templates that are used to send emails to users.
You can customize these templates to your liking.
You will find the templates in Nile Console when you navigate to **Tenants and Users**, click on
the **Configure** tab and then on **Email Templates**.
There are 2 available templates:
## Email invitation
Sent when a user is invited to join a tenant, or uses their email address as the sign in provider.
## Reset password
Sent when a user requests a password reset. This is automatically triggered with a `POST` request to `/api/auth/reset-password`
The `callbackURL` sent to the endpoint will be modified with the token in the **api\_url** variable, or it will fall back to the `niledb-origin` header, which is automatically set via the JS SDK
## Customizing the templates
The templates are fully customizable with HTML and CSS. You can use the available variables
(shown on the right side of the template editor) to customize the email content, and you can also add variables of your own.
For your convenience, you can preview the email that the template will generate:
## Related Topics
* [Email Verification](/auth/email/verification)
* [Custom SMTP](/auth/email/customsmtp)
* [Per-tenant Overrides](/auth/concepts/per-tenantoverrides)
# null
Source: https://thenile.dev/docs/auth/email/verification
## Overview
Email verification helps confirm user identities by requiring them to verify their email addresses before gaining full access. This is especially useful for preventing spam and unauthorized access.
When a user signs up, Nile Auth:
1. Sends a verification email using your configured SMTP server.
2. Generates a unique `verification_token` for the user.
3. Grants access to the user after they click the verification link.
## Prerequisites
* A configured SMTP server is required to send verification emails. You can set this up in the Nile Auth UI under **Tenants & Users -> Configuration -> Email templates**.
* A web server configured to handle Nile Auth requests and serve html.
## Implementation
```bash theme={null}
npm install @niledatabase/server @niledatabase/react
@niledatabase/client @niledatabase/nextjs
```
Your application must expose API routes to handle authentication operations.
Create a folder called `api` under the `app` folder and a folder called `[...nile]` under it:
```bash theme={null}
mkdir -p app/api/\[...nile\]
```
Create following files handle the calls to your server, as well as expose the `nile` instance to your application:
```typescript app/api/[...nile]/nile.ts theme={null}
import { Nile } from '@niledatabase/server';
import { nextJs } from '@niledatabase/nextjs';
export const nile = Nile({ extensions: [nextJs] });
```
```typescript app/api/[...nile]/route.ts theme={null}
import { nile } from './nile';
export const { POST, GET, DELETE, PUT } = nile.handlers;
```
```typescript theme={null}
import { Nile } from "@niledatabase/server";
export const nile = Nile();
app.use("verification-page", async (req, res, next) => {
const me = await nile.users.getSelf(req.headers);
if (!me.emailVerified) {
return res.redirect("/force-verification");
}
next();
});
```
This middleware ensures that unverified users are redirected to a verification page.
```typescript theme={null}
"use client";
import { User } from "@niledatabase/server";
import { EmailSignInButton } from "@niledatabase/react";
export default function VerifyButton({ me }: { me: User }) {
return ;
}
```
This button allows users to trigger email verification manually.
You can also allow users to sign in directly using their email without requiring a password.
```typescript theme={null}
"use client";
import { EmailSignIn } from "@niledatabase/react";
export default function EmailLogin() {
return ;
}
```
## **Verification flow**
1. The user inputs their email on the login/signup page.
2. Nile sends an email with a verification token using the configured SMTP settings.
3. The link exchanges the `verification_token` for an authenticated session.
4. Upon successful verification, the user can access the application.
## **Related Topics**
* [Email Templates](/auth/email/templates) - Customize email verification messages.
* [Custom SMTP](/auth/email/customsmtp) - Configure your own SMTP server for sending emails.
* [User Management](/auth/concepts/users) - Manage user accounts in Nile Auth.
# Examples
Source: https://thenile.dev/docs/auth/examples
End to end examples for adding Nile Auth to your B2B application
These are end to end examples that you can fork and use it to build your own applications. It works with Nile's Postgres and Nile Auth.
## Workflows
}
>
Login with Email or Magic Link
}
>
Verify user using email
}
>
Initiate password reset flow
}
>
Add Google SSO support
## Frameworks
}
>
Auth with NextJS
}
>
Auth with Node
}
>
Auth with Remix
}
>
Auth with Drizzle
{"file_type_prisma"}
}
>
Auth with Prisma
# Elysia
Source: https://thenile.dev/docs/auth/frameworks/elysia
Integrate Nile Auth with Elysia applications
This guide will help you get started with Nile Auth and ElysiaJS. This guide outlines the steps required to configure and integrate authentication in your
application.
If you have not done so yet, be sure you have [obtained credentials from the
console](../getting-started/installation).
```bash bun theme={null}
bun add @niledatabase/server @niledatabase/elysia elysia
```
The simplest way to use Nile with Elysia is to register the plugin. This will automatically mount Nile's default API routes ([authentication](/reference/api/auth), [tenant management](/reference/api/tenants), etc.) onto your Elysia application.
```typescript app.ts theme={null}
import { Elysia } from 'elysia';
import { Nile } from '@niledatabase/server';
import { nilePlugin } from '@niledatabase/elysia';
// Initialize the Nile Server
const nile = Nile({
debug: true,
// ... configuration options
});
const app = new Elysia().use(nilePlugin(nile));
// The plugin has now registered routes like:
// POST /api/auth/signin
// POST /api/auth/signup
// GET /api/auth/session
// ...and more
app.listen(3000);
console.log(
`🦊 Elysia is running at ${app.server?.hostname}:${app.server?.port}`,
);
```
## Accessing the Nile Instance
The `nilePlugin` automatically decorates the Elysia context with your `nile` server instance. This allows you to access it directly in your handlers for database queries or custom auth logic.
```typescript theme={null}
app.get('/my-data', async ({ nile, request, error }) => {
// 1. You can access the configured nile instance directly
const headers = new Headers(request.headers);
const tenantId = request.headers.get('X-Tenant-Id') || undefined;
// 2. Use nile.withContext to ensure isolation
return await nile.withContext({ headers, tenantId }, async (nileCtx) => {
// This query will automatically apply the tenantId for the tenant-aware table
const result = await nileCtx.db.query('SELECT * FROM my_table');
return result.rows;
});
});
```
## Deriving the User
For a more idiomatic Elysia experience, you can create a derivation that extracts the authenticated user from the context.
```typescript theme={null}
const authApp = app.derive(async ({ nile, request }) => {
const headers = new Headers(request.headers);
// Use the decorated nile instance to fetch the session
const user = await nile.withContext({ headers }, async (nileCtx) => {
return await nileCtx.auth.getSession();
});
return { user };
});
authApp.get('/profile', ({ user, error }) => {
if (!user) {
return error(401, 'Not Authenticated');
}
return user;
});
```
## Custom Auth Helpers
You can extend the derivation pattern to expose helper methods that automatically handle context, making your handlers cleaner.
```typescript theme={null}
// Create a plugin that derives the user and auth helpers for every request
const authPlugin = (app: Elysia) =>
app.derive(async ({ nile, request }) => {
// 1. Create headers from the incoming request to propagate cookies/auth
const headers = new Headers(request.headers);
// 2. Use the decorated nile instance to get the session
const user = await nile.withContext({ headers }, async (nileCtx) => {
return await nileCtx.auth.getSession();
});
return {
Auth: {
user,
// Wrap SDK methods to automatically provide context
signUp: async (email: string, password: string) => {
return nile.withContext({ headers }, (c) =>
c.auth.signUp({ email, password }),
);
},
signIn: async (email: string, password: string) => {
return nile.withContext({ headers }, (c) =>
c.auth.signIn('credentials', { email, password }),
);
},
},
};
});
```
Now you can use the plugin in your application and access `Auth.user` and methods directly.
```typescript theme={null}
app
.use(authPlugin)
.get('/profile', ({ Auth: { user }, error }) => {
if (!user) {
return error(401, 'Not Authenticated');
}
return user;
})
.post('/register', async ({ Auth, body }) => {
const { email, password } = body as any;
return await Auth.signUp(email, password);
});
```
## The Plugin
Under the hood, the elysia plugin does the following:
1. **Registers Routes**: Maps Nile's internal route handlers to Elysia's router (e.g. `app.get`, `app.post`), exposing the full API.
2. **Decorates Context**: Adds the `nile` server instance to the Elysia context (`ctx.nile`), making it available in all downstream handlers and plugins.
3. **Context Isolation**: For the routes it registers, it wraps execution in `nile.withContext` to ensure tenant isolation and proper authentication handling.
### Handlers and paths
The plugin iterates over the configured paths in the Nile Server SDK and registers them directly with your Elysia app instance. This ensures that all standard [authentication](/reference/api/auth) and [tenant management](/reference/api/tenants) routes are available without manual configuration.
# Express
Source: https://thenile.dev/docs/auth/frameworks/express
Integrate Nile Auth with express applications
This guide will help you get started with Nile Auth and Express. This guide outlines the steps required to configure and integrate authentication in your
application. This guide assumes that you have already created an Express application, if you haven't, you can use the [Quickstart](/auth/quickstarts/express) to get started.
If you have not done so yet, be sure you have [obtained credentials from the
console](../getting-started/installation).
```bash npm theme={null}
npm install @niledatabase/server @niledatabase/express
```
```bash yarn theme={null}
yarn add @niledatabase/server @niledatbase/express
```
```bash pnpm theme={null}
pnpm add @niledatabase/server @niledatabase/express
```
Out of the box, the express extension will add all routes and middleware to your express application to ensure everything works properly.
```ts app/api/[...nile]/route.ts theme={null}
import { express } from 'express';
import { Nile } from '@niledatabase/server';
import { express as nileExpress } from '@niledatabase/express';
const app = express();
export const nile = Nile({
extensions: [nileExpress(app)],
origin: fe_url, // whitelist your FE origin
});
```
## The extension
Under the hood, the express extension does the following:
1. Reconfigures `handlers[method]` and `paths[method]` to be compatible with express.
2. Modifies the request handling to match express
3. Uses a combination of `AsyncLocalStorage` and middleware to ensure each request has the correct context.
### Handlers and paths
The express extension registers routes though the specific methods, not as middleware, because nile-auth can span a lot of routes (`/api/auth/signin`, `/api/tenants/:tenantId/users`, etc). This allows you to override specific routes, or add new ones since nile only matches specific routes (either the default, or ones that have been configured)
Additionally, since the base `@niledatabase/server` assumes standard `Request` and `Response` objects, this extension turns express objects into those and responds accordingly.
Lastly, because of the express lifecycle, its important to be sure that each requests is executed within a specific context. We automatically add a `:tenantId` param listener and set the context of the nile instance to that tenantId for convenience. That way, no matter the request or DB query, if you have the tenantId in your param (this also works automatically with the `nile.tenantId` cookie), there's no additional work that needs done.
# Next.js
Source: https://thenile.dev/docs/auth/frameworks/nextjs
Integrate Nile Auth with Next.js applications
This guide will help you get started with Nile Auth and NextJS. This guide outlines the steps required to configure and integrate authentication in your
application. This guide assumes that you have already created a NextJS application, if you haven't, you can use the [Quickstart](/auth/quickstarts/nextjs) to get started.
If you have not done so yet, be sure you have [obtained credentials from the
console](../getting-started/installation).
```bash npm theme={null}
npm install @niledatabase/server @niledatabase/nextjs @niledatabase/react @niledatabase/client
```
```bash yarn theme={null}
yarn add @niledatabase/server @niledatbase/nextjs @niledatabase/react @niledatabase/client
```
```bash pnpm theme={null}
pnpm add @niledatabase/server @niledatabase/nextjs @niledatabase/react @niledatabase/client
```
Configure the splat route. Be sure to add the nextJs extension, it will handle passing the client requests (`next/headers`), as well passing the correct headers on server-side calls automatically.
```typescript app/api/[...nile]/nile.ts theme={null}
import { Nile } from '@niledatabase/server';
import { nextJs } from '@niledatabase/nextjs';
export const nile = Nile({ extensions: [nextJs] });
```
```typescript app/api/[...nile]/route.ts theme={null}
import { nile } from './nile';
export const { POST, GET, DELETE, PUT } = nile.handlers;
```
## Server actions and RSC
The build-in components found in `@niledatabase/react` will request information client side. If you'd like to hydrate them, `@niledatabase/server` functions can be called.
### Example: Verify email address
```ts app/verify-email/page.tsx theme={null}
import { UserInfo } from "@niledatabase/react";
import { nile } from "../api/[...nile]/nile";
export default async function VerifyEmail() {
const me = await nile.users.getSelf(); // either the json parsed user, or a request (401)
if (me instanceof Response) {
return "You must be logged in.";
}
return (
// rest of component, probably a form with a button that calls the `action`
below
);
}
```
Server actions work in a similar way. Here's an example of an action that is called to verify a user's email address. An action like this requires SMTP to be configured in the console.
```ts theme={null}
async function action() {
'use server';
const res = await nile.users.verifySelf(
{
callbackUrl: '/dashboard', // customize where the user should land after their email is verified
},
true,
);
if (!res.ok) {
return { ok: false, message: await res.text() }; // errors are text
}
return { ok: true, message: 'Check your email for your verification' };
}
```
`nile.users.verifySelf` mirrors the client-side call to
`/api/auth/verify-email`. All server side calls can be called from a client
based on their configured route.
### Client side
While in most cases, actions would be used to handle the calls to nile-auth, it is possible to use `@niledatabase/react` which can do a lot of the heavy lifting for you, especially when it comes to client-side authorization, or quickly getting a demo app up and running.
### Example: Hydrated sign up / dashboard page
```ts theme={null}
import {
SignOutButton,
SignUpForm,
SignedIn,
SignedOut,
TenantSelector,
UserInfo,
} from "@niledatabase/react";
import { nile } from "../api/[...nile]/nile";
import { Tenant, User } from "@niledatabase/server";
export default async function SignUpPage() {
// be sure nothing happens on the edge
const [session, me, tenants] = await nile.withContext(async (ctx) => Promise.all([
ctx.auth.getSession(),
ctx.users.getSelf(),
ctx.tenants.list(),
]);
return (
);
}
```
For something like SSO, the easiest thing to do is create a button for signin in, and `@niledatabase/client` to handle the rest
```ts theme={null}
"use client";
import { Button } from "@/components/ui/button";
import { signIn, getSession } from "@niledatabase/client";
export default function AllClient() {
const session = getSession();
return (
Session data: {session ? JSON.stringify(session) : null}
{
signIn("google", { callbackUrl: "/allClientSide" });
}}
>
Sign in to google
);
}
```
## Related topics
* [SDK reference](../sdk-reference/javascript/overview)
# Nitro
Source: https://thenile.dev/docs/auth/frameworks/nitro
Integrate Nile Auth with Nitro applications
This guide will help you get started with Nile Auth and Nitro. This guide outlines the steps required to configure and integrate authentication in your application. This guide uses the `Nuxt` layouts, and can be adjusted for any `Nitro` application
If you have not done so yet, be sure you have [obtained credentials from the
console](../getting-started/installation).
```bash npm theme={null}
npm install @niledatabase/server @niledatabase/nitro
```
```bash yarn theme={null}
yarn add @niledatabase/server @niledatbase/nitro
```
```bash pnpm theme={null}
pnpm add @niledatabase/server @niledatabase/nitro
```
Create the base nile instance, extended with the nitro plugin
```ts app/composables/useNile.ts theme={null}
import { Nile } from "@niledatabase/server";
import { nitro } from "@niledatabase/nitro";
import type { withNitro } from "@niledatabase/nitro";
export const nile = Nile({
debug: true,
extensions: [nitro],
}));
```
Add the route so that the APIs can be handled `ts server/api/[...slug].ts
import {nile} from "~/composables/useNile"; export default
defineEventHandler(nile.handlers); `
## The extension
Under the hood, the nitro extension does the following reconfigures `handlers` to use the h3 callbacks, vs the default. The default method calls are still handled the same way internally, it only modifies the base call.
For instance, the calls on the server-side will still need to have their context set.
```ts theme={null}
import { nile } from '~/composables/useNile';
const serverFunction = (event) => {
const ctxNile = nile.withContext({ headers: event.headers });
const userTenants = await ctxNile.tenants.list();
};
```
# Nuxt
Source: https://thenile.dev/docs/auth/frameworks/nuxt
Integrate Nile Auth with Nuxt applications
This guide will help you get started with Nile Auth and Nuxt. This guide outlines the steps required to configure and integrate authentication in your application.
If you have not done so yet, be sure you have [obtained credentials from the
console](../getting-started/installation).
```bash npm theme={null}
npm install @niledatabase/server @niledatabase/nitro @niledatabase/client
```
```bash yarn theme={null}
yarn add @niledatabase/server @niledatbase/nitro @niledatabase/client
```
```bash pnpm theme={null}
pnpm add @niledatabase/server @niledatabase/nitro @niledatabase/client
```
Create the base nile instance, extended with the nitro plugin
```ts app/composables/useNile.ts theme={null}
import { Nile } from "@niledatabase/server";
import { nitro } from "@niledatabase/nitro";
import type { withNitro } from "@niledatabase/nitro";
export const nile = Nile({
debug: true,
extensions: [nitro],
}));
```
Add the route so that the APIs can be handled `ts server/api/[...slug].ts
import {nile} from "~/composables/useNile"; export default
defineEventHandler(nile.handlers); `
## Client side
For requests in Vue, you can do the following
```Vue theme={null}
<-- Sign in form here -->
```
Not every API call is supported in the client. You may need to call some routes via the framework `useFetch('/api/tenants')`, or `defineEventHandler`
```ts server/api/submit-invite.post.ts theme={null}
export default defineEventHandler(async (event) => {
const ctxNile = nile.withContext({ headers: event.headers });
const body = await readBody(event);
const res = ctxNile.tenants.invite({ email: body.email });
if (res instanceof Response) {
return { message: 'An error has occurred' };
}
return { message: 'Invite sent' };
});
```
# React
Source: https://thenile.dev/docs/auth/frameworks/react
Integrate Nile Auth with React applications
## Installation
`@niledatabase/react` contains components and hooks for using the API. It handles sessions, cookies, fetching data, and comes with built-in components to be used as either templates or directly in your application to handle common user tasks. It is designed to be used with `@niledatabase/server`, which handles API calls itself or forwards them to the regional API for your database.
```bash theme={null}
npm install @niledatabase/react @niledatabase/client
```
## Using the Auth Provider
`@niledatabase/react` comes with two providers, ` ` and ` ` which wrap a central ` `. By default, they will fetch a session.
` ` will only render children if the user is logged in. Conversely, ` ` will always render its children unless signed in.
```jsx theme={null}
Jerry, hello!
```
```jsx theme={null}
No soup for you!
```
It is also possible to be explicit and obtain the session server side. To do that, you would use the following:
```jsx theme={null}
import { SignedIn } from '@niledatabase/react';
import { nile } from '@/lib/nile';
export default async function SignUpPage() {
const nileCtx = nile.withContext({ headers }); // headers from request
const session = await nileCtx.auth.getSession();
if (session.status === 'unauthenticated') {
return No soup for you!
;
}
return Jerry, hello! ;
}
```
## Functions
### signIn
makes a `POST` request to the sign in endpoint. Expects a provider, and optional params for `callbackUrl`. For the cases of `credentials` (email + password) and `email`, you can opt out of redirection by passing `redirect: false` in the options
```jsx theme={null}
signIn('google', { callbackUrl: '/dashboard' });
signIn('credentials', { callbackUrl: '/dashboard' });
```
### signOut
makes a `POST` request to the sign out endpoint, with an optional params for `callbackUrl` to redirect the user, and `redirect` to leave the user on the page, but delete the session and notify the session providers the user has been logged out.
```jsx theme={null}
signOut({ callbackUrl: '/sign-in' }); // go back to some sign in page
signOut({ redirect: false }); // Log out, leave the user on the page they're on
```
## Hooks
### useSession
You can obtain the current session via `useSession()`. This must be called within a ` ` or ` ` provider.
```jsx theme={null}
import { useSession, UserInfo } from '@niledatabase/react';
export default function SignUpPage() {
const session = useSession();
if (session.status !== 'authenticated') {
return Loading...
;
}
return ;
}
```
### useTenantId
The `useTenantId` hook manages the current tenant ID, persisting it in cookies and refetching tenant data when necessary. A tenant id is accessible via `document.cookie` with the name `nile.tenant_id`. This cookie is used by the server side SDK to make requests to the auth service.
```tsx theme={null}
import { useTenantId } from '@niledatabase/react';
export default function TenantSelector() {
const [tenantId, setTenantId] = useTenantId();
return (
Current Tenant: {tenantId ?? 'None'}
setTenantId('new-tenant-id')}>
Change Tenant
);
}
```
| Value | Type | Description |
| ------------- | -------------------------- | --------------------------------- |
| `tenantId` | `string \| undefined` | The current tenant ID. |
| `setTenantId` | `(tenant: string) => void` | Function to update the tenant ID. |
| Name | Type | Default | Description |
| -------- | -------------------------------- | ----------- | ---------------------------- |
| `params` | `HookProps & { tenant: Tenant }` | `undefined` | Initial tenant data. |
| `client` | `QueryClient` | `undefined` | React Query client instance. |
```ts theme={null}
export type HookProps = {
tenants?: Tenant[]; // a list of tenants
onError?: (e: Error) => void; // failure callback
baseUrl?: string; // fetch origin
};
```
* Initializes the tenant ID from `params.tenant.id`, if provided.
* If no tenant is found, it attempts to read from a cookie (`nile.tenant_id`).
* If no cookie exists, it triggers a refetch of tenants.
* Calling `setTenantId(tenantId)` updates both state and cookie.
### useTenants
The `useTenants` hook fetches a list of tenants from an API endpoint using React Query. It supports optional preloaded data and can be disabled to prevent automatic queries.
```tsx theme={null}
import { useTenants } from '@niledatabase/react';
export default function TenantList() {
const { data: tenants, isLoading, error } = useTenants();
if (isLoading) return Loading tenants...
;
if (error) return Error loading tenants
;
return (
{tenants?.map((tenant) => (
{tenant.name}
))}
);
}
```
This hook returns the result of `useQuery`, which includes:
| Property | Type | Description |
| ----------- | ----------------------- | ------------------------------------- |
| `data` | `Tenant[] \| undefined` | List of tenants. |
| `isLoading` | `boolean` | `true` while fetching data. |
| `error` | `Error \| null` | Fetch error, if any. |
| `refetch` | `() => void` | Function to manually refetch tenants. |
**Parameters**
| Name | Type | Default | Description |
| -------- | ---------------------------------------- | ----------- | ------------------------------------- |
| `params` | `HookProps & { disableQuery?: boolean }` | `undefined` | Hook configuration options. |
| `client` | `QueryClient` | `undefined` | Optional React Query client instance. |
```ts theme={null}
export type HookProps = {
tenants?: Tenant[];
onError?: (e: Error) => void;
baseUrl?: string;
};
```
* If `disableQuery` is `true`, the query is disabled.
* If `tenants` is provided and not empty (in the event of hydration), the query is also disabled.
* Otherwise, it fetches tenants from `${baseUrl}/api/tenants` using `fetch`.
* The request runs **only once** unless manually refetched.
### useEmailSignIn
The `useEmailSignIn` hook provides a mutation for signing in a user using nile auth. It allows customizing the request with callbacks and options for redirection.
```tsx theme={null}
import { useSignIn } from '@your-library/react';
export default function Login() {
const signIn = useEmailSignIn({
onSuccess: () => console.log('Login successful'),
onError: (error) => console.error('Login failed', error),
redirect: true,
});
return (
signIn({ email: 'user@example.com' })}>
Sign In
);
}
```
| Name | Type | Default | Description |
| -------------- | -------------------------- | ----------- | ---------------------------------------- |
| `onSuccess` | `(data: Response) => void` | `undefined` | Callback after a successful sign-in. |
| `onError` | `(error: Error) => void` | `undefined` | Callback if sign-in fails. |
| `beforeMutate` | `(data: any) => any` | `undefined` | Function to modify data before mutation. |
| `callbackUrl` | `string` | `undefined` | URL to redirect after login. |
| `redirect` | `boolean` | `false` | Whether to redirect after login. |
* Calls `signIn('email', data)` with optional modifications via `beforeMutate`.
* Throws an error if authentication fails.
* Redirects if `redirect` is `true`.
### useMe
`useMe` is a React hook that fetches and returns the current authenticated user. It allows preloading a user via props or fetching from an API endpoint if no user is provided.
```tsx theme={null}
'use client';
import { useMe } from '@niledatabase/react';
export default function Profile() {
const user = useMe();
if (!user) return Loading...
;
return Welcome, {user.name}!
;
}
```
| Name | Type | Default | Description |
| ---------- | --------------------------- | ----------- | ----------------------------------------------- |
| `fetchUrl` | `string` | `/api/me` | API endpoint to fetch the user data. |
| `user` | `User \| undefined \| null` | `undefined` | Initial user data, avoids fetching if provided. |
* If a `user` is passed in props, it is set immediately.
* If `user` is not provided, the hook fetches from `fetchUrl` and updates the state.
* The request runs **only once** when the component mounts.
### useResetPassword
The `useResetPassword` hook provides a way to handle password reset functionality. It sends reset requests to an authentication API and supports optional callbacks and preprocessing of request data.
```tsx theme={null}
import { useResetPassword } from '@niledatabase/react';
export default function ResetPasswordForm() {
const resetPassword = useResetPassword({
onSuccess: () => alert('Password reset successful'),
onError: (err) => console.error('Error resetting password:', err),
});
const handleSubmit = (e: React.FormEvent) => {
e.preventDefault();
const formData = new FormData(e.target as HTMLFormElement);
resetPassword({
email: formData.get('email') as string,
});
};
return (
);
}
```
| Name | Type | Default | Description |
| -------- | -------- | ----------- | --------------------------- |
| `params` | `Params` | `undefined` | Hook configuration options. |
```ts theme={null}
export type Params = {
onSuccess?: (data: Response) => void;
onError?: (error: Error) => void;
beforeMutate?: (data: MutateFnParams) => MutateFnParams;
callbackUrl?: string;
baseUrl?: string;
fetchUrl?: string;
};
```
* Calls the API at `${baseUrl}/api/auth/reset-password` (or a custom `fetchUrl`).
* Uses **PUT** for password updates and **POST** for reset requests.
* Calls `onSuccess` or `onError` based on the request outcome.
* Runs a CSRF request when the hook is initialized.
* Allows modifying data before sending using `beforeMutate`.
### useSignUp
The `useSignUp` hook provides a way to handle user sign-up requests. It supports tenant creation, API customization, and session updates after successful registration.
```tsx theme={null}
import { useSignUp } from '@niledatabase/react';
export default function SignUpForm() {
const signUp = useSignUp({
onSuccess: () => alert('Sign-up successful!'),
onError: (err) => console.error('Sign-up failed:', err),
createTenant: true, // Optionally create a tenant with the email of the user
});
const handleSubmit = (e: React.FormEvent) => {
e.preventDefault();
const formData = new FormData(e.target as HTMLFormElement);
signUp({
email: formData.get('email') as string,
password: formData.get('password') as string,
});
};
return (
);
}
```
| Name | Type | Default | Description |
| -------- | ------------------------ | ----------- | ---------------------------- |
| `params` | `Props` | `undefined` | Hook configuration options. |
| `client` | `QueryClient` (optional) | `undefined` | React Query client instance. |
```ts theme={null}
export type Props = {
onSuccess?: (data: Response, variables: unknown) => void; // success callback
onError?: (error: Error) => void; // error callback
beforeMutate?: (data: SignUpInfo) => SignUpInfo; // optional modifications to the data prior to the API call
callbackUrl?: string; // where the server should redirect users upon successful login
baseUrl?: string; // configure fetch origin
createTenant?: boolean | string; // if boolean, will create a tenant named with the user's email, else will be whatever name is provided (maps to /api/tenants?newTenantName=)
};
```
```ts theme={null}
export type SignUpInfo = {
email: string;
password: string;
tenantId?: string;
newTenantName?: string;
fetchUrl?: string;
};
```
* Sends a `POST` request to `/api/signup` (or a custom `fetchUrl`).
* If `createTenant` is `true`, assigns `newTenantName` as the user’s email.
* If `createTenant` is a string, it is used as the tenant name.
* After a successful sign-up:
* Updates the session.
* Redirects to `callbackUrl` if provided, otherwise reloads the page.
* Prefetches authentication providers and CSRF tokens on mount.
### useSignIn
The `useSignIn` hook provides a simple way to authenticate users. It supports pre-processing login data before submission and handles authentication via credentials.
```tsx theme={null}
import { useSignIn } from '@niledatabase/react';
export default function SignInForm() {
const signIn = useSignIn({
onSuccess: () => alert('Login successful!'),
onError: (err) => console.error('Login failed:', err),
});
const handleSubmit = (e: React.FormEvent) => {
e.preventDefault();
const formData = new FormData(e.target as HTMLFormElement);
signIn({
email: formData.get('email') as string,
password: formData.get('password') as string,
});
};
return (
);
}
```
| Name | Type | Default | Description |
| -------- | ------- | ----------- | --------------------------- |
| `params` | `Props` | `undefined` | Hook configuration options. |
```ts theme={null}
export type Props = {
onSuccess?: () => void;
onError?: (error: Error) => void;
beforeMutate?: (data: LoginInfo) => LoginInfo;
callbackUrl?: string;
};
```
```ts theme={null}
export type LoginInfo = {
email: string;
password: string;
};
```
***
* Sends a sign-in request using NextAuth's `signIn('credentials', data)`.
* If `beforeMutate` is provided, it modifies the login data before the request.
* Calls `onSuccess` if login succeeds.
* Calls `onError` if login fails.
## Multi-factor (Client)
Client-side MFA enrollment, challenge, and removal flows powered by the React SDK. The `User` object (obtained via `useSession`) will include a `multiFactor` property when MFA is enabled for the user.
### Features
* Authenticator app or email one-time-code enrollment against `/auth/mfa`
* Recovery codes for authenticator challenges with remaining-count feedback
* Redirect-aware helpers that work with custom routers or default navigation (`ChallengeRedirect`)
* UI building blocks (`MultiFactor*` components) plus a lightweight hook
* Optional low-level `mfa` helper for bespoke prompts or headless flows
### Overview
Nile Auth exposes a single `/auth/mfa` endpoint for starting MFA setup, completing challenges, and removing an enrolled method. The React SDK wraps the endpoint in `useMultiFactor` and UI components that parse server responses into ready-to-render experiences.
Tokens returned from setup or sign-in responses must be echoed back on subsequent calls (challenge verification or removal). When the backend needs the browser to redirect, the helper returns `{ url: string }` (a `ChallengeRedirect`) so you can route accordingly.
### Installation
```sh theme={null}
npm install @niledatabase/react @niledatabase/client
```
### Quick start
#### Enroll with an authenticator app
```tsx theme={null}
import { useMultiFactor, MultiFactorAuthenticator } from '@niledatabase/react';
export function MfaEnrollment() {
const { setup, loading, startSetup } = useMultiFactor({
method: 'authenticator',
currentMethod: null,
onRedirect: (url) => window.location.assign(url), // optional
});
return (
<>
{loading ? 'Starting...' : 'Enable authenticator MFA'}
{setup?.scope === 'setup' && setup.method === 'authenticator' ? (
scope === 'setup' && window.location.reload()}
/>
) : null}
>
);
}
```
#### Handle a challenge prompt (sign-in or removal)
```tsx theme={null}
import { useMultiFactor, MultiFactorChallenge } from '@niledatabase/react';
export function ChallengePrompt({ existingToken }: { existingToken: string }) {
const { setup, startDisable } = useMultiFactor({
method: 'authenticator',
currentMethod: 'authenticator',
});
return (
<>
{/* Example: Disabling MFA requires a challenge verification first */}
Disable MFA
{setup?.scope === 'challenge' ? (
window.location.replace('/app')}
/>
) : null}
>
);
}
```
### API
#### `useMultiFactor(options)`
| Name | Type | Default | Description |
| --------------------- | ---------------------------------------------------------- | --------------------------------- | --------------------------------------------------------------------------------------- |
| `method` | `'authenticator' \| 'email'` | *(required)* | MFA mechanism to enable or disable. |
| `currentMethod` | `'authenticator' \| 'email' \| null` | `null` | Currently enrolled method; blocks switching until disabled. |
| `onRedirect` | `(url: string) => void` | `window.location.assign` | Optional handler when the backend responds with a `url` to visit (`ChallengeRedirect`). |
| `onChallengeRedirect` | `(params: { token; method; scope; destination? }) => void` | Internal `/mfa/prompt` navigation | Override the default challenge redirect builder. |
Returns `{ setup, loading, errorType, startSetup, startDisable }`.
* `setup`: `null` or an MFA payload. For authenticator: `{ method: 'authenticator'; token; scope; otpauthUrl?; secret?; recoveryKeys? }`. For email: `{ method: 'email'; token; scope; maskedEmail? }`.
* `startSetup()`: begins enrollment (POST `/auth/mfa`); `setup.scope` will be `"setup"` or `"challenge"` if a verification step is required.
* `startDisable()`: starts removal for the given method. If verification is required, `setup.scope` will be `"challenge"`.
* `errorType`: one of `setup`, `disable`, `parseSetup`, `parseDisable`, or `null` for success.
### Components
* `MultiFactorAuthenticator` — renders QR code, recovery keys, and a verification form.\
Props: `setup: AuthenticatorSetup`, `onError(message: string | null)`, `onSuccess(scope: 'setup' | 'challenge')`.
* `MultiFactorEmail` — shows masked email messaging and a verification form.\
Props: `setup: EmailSetup`, `onSuccess(scope: 'setup' | 'challenge')`.
* `MultiFactorChallenge` — shared challenge UI for either method (used for disable flows or sign-in prompts).\
Props: `payload: { token: string; scope: ChallengeScope; method: MfaMethod }`, `message: string`, `isEnrolled: boolean`, `onSuccess(scope)`.
### Error handling
* Non-200 responses are coerced into `{ url: string }` (`ChallengeRedirect`) with an `error` search param. `MfaVerifyForm` and the examples above parse this for user-friendly messaging.
* If `code` is missing or shorter than 6 digits, the React forms set a validation error before calling the API.
### Additional notes
* Codes are expected to be 6 digits for authenticator/email verification; recovery codes are string tokens issued during setup.
* Challenge tokens expire; expect 410 responses for stale tokens and 404 for unknown challenges.
* Email MFA may return a `maskedEmail` and require the same token on verification and disable flows.
## Related Topics
* [MFA Concepts](/auth/concepts/multifactor)
* [Next.js Integration](/auth/frameworks/nextjs)
* [Components](/auth/components/signin)
# Remix
Source: https://thenile.dev/docs/auth/frameworks/remix
Integrate Nile Auth with Remix applications
This guide explains how to integrate **Nile Database** with **Remix** and set up routes for handling various HTTP requests (`GET`, `POST`, `PUT`, `DELETE`). Additionally, you'll see how to include **client-side components** for user authentication and interaction using **Nile's React SDK**.
This guide assumes you are running an application similar to `npx
create-react-router@latest --template
remix-run/react-router-templates/node-postgres`, and have already updated your
`.env` file from [the installation](../getting-started/installation.mdx)
***
Update your .env file with `DATABASE_URL` so drizzle works.
```bash .env theme={null}
DATABASE_URL=postgres://niledb_user:niledb_password0@us-west-2.db.thenile.dev:5432/
```
Update `/server/app.ts`
```ts server/app.ts theme={null}
import postgres from 'pg';
```
Along with that change, sync up `database/context.ts` to be sure its using the same types, changing from `PostgresJsDatabase`, to `NodePgDatabase`. In addition, there are may be some compatibility changes that need handled if you are updating an existing project [based on our compatibility](../../postgres/postgres-compatibility)
```ts theme={null}
import type { NodePgDatabase } from 'drizzle-orm/node-postgres';
export const DatabaseContext = new AsyncLocalStorage<
NodePgDatabase
>();
```
Now we need to add the nile instance and route handlers to allow our server to respond to authentication, user, and tenant requests.
```ts app/routes/api.$.ts theme={null}
import type { LoaderFunctionArgs, ActionFunctionArgs } from '@remix-run/node';
import { nile } from '~/nile'; // the nile instance
export const loader = async ({ request }: LoaderFunctionArgs) => {
const method = request.method.toUpperCase();
return nile.handlers[method](request);
};
export const action = async ({ request }: ActionFunctionArgs) => {
const method = request.method.toUpperCase();
return nile.handlers[method](request);
};
```
Then update your route config to allow the handlers to work
```ts app/routes.ts theme={null}
import { type RouteConfig, index, route } from '@react-router/dev/routes';
export default [
index('routes/home.tsx'),
route('api/*', 'app/routes/api.$.ts'),
] satisfies RouteConfig;
```
## Creating a loader
In some cases, you may want to create specific action and loader around the API, to do that, use the server side functions in the sdk in your loader. This code loads and updates a user's profile.
```tsx theme={null}
import type { Route } from "./+types/profile";
import { nile } from "~/nile";
export const loader: LoaderFunction = async ({ request }) => {
try {
const _nile = nile.withContext({ headers: request.headers });// be sure the context is correct from the request
const user = await _nile.users.getSelf();
if (user) {
// If the user is authenticated, we can return their info or pass it to the UI
return json({ user });
} else {
// If the user is not authenticated, redirect to the index page
return redirect("/");
}
} catch (error) {
return json({ message: error.message }, { status: 500 });
}
};
export default function Profile({ loaderData }: : Route.ComponentProps) {
const { user, message } = loaderData;
return (
{message ? <>{message}> : null}
);
}
```
## Related Topics
* [React Integration](/auth/frameworks/react)
* [Components](/auth/components/signin)
# Installation
Source: https://thenile.dev/docs/auth/getting-started/installation
Install nile-auth in your application
```bash npm theme={null}
npm install @niledatabase/server @niledatabase/client
```
```bash yarn theme={null}
yarn add @niledatabase/server @niledatabase/client
```
```bash pnpm theme={null}
pnpm add @niledatabase/server @niledatabase/client
```
1. If you haven't signed up for Nile yet, [sign up here](https://console.thenile.dev) and follow the steps to create a database.
2. Navigate to **Database Settings** in your database's UI at [console.thenile.dev](https://console.thenile.dev).
3. Go to **Connection** settings.
4. Select the CLI icon, and click **Generate credentials**
5. **Copy** the required credentials and **store them in an `.env` file** so they can be used in the application to connect to the Nile auth service.
```bash .env theme={null}
NILEDB_USER=niledb_user
NILEDB_PASSWORD=niledb_password
NILEDB_API_URL=https://us-west-2.api.thenile.dev/v2/databases/
NILEDB_POSTGRES_URL=postgres://us-west-2.db.thenile.dev:5432/
```
```ts nextjs theme={null}
import { Nile } from '@niledatabase/server';
import { nextJs } from '@niledatabase/nextjs';
export const nile = Nile({ extensions: [nextJs] });
```
```ts remix theme={null}
import { Nile } from '@niledatabase/server';
export const nile = Nile();
```
```ts express theme={null}
import { Nile } from '@niledatabase/server';
import { express } from '@niledatabase/express';
export const nile = Nile({ extensions: [express(app)] });
```
```ts nitro theme={null}
import { Nile } from '@niledatabase/server';
import { nitro } from '@niledatabase/nitro ';
export const nile = Nile({ extensions: [nitro] });
```
```ts JS theme={null}
import { Nile } from '@niledatabase/server';
export const nile = Nile();
```
To handle requests, set up a route handler on your server.
```ts next-js theme={null}
// app/api/[...nile]/route.ts
import { nile } from "@/nile"; // path to nile instance
export const { POST, GET, DELETE, PUT } = nile.handlers;
```
```ts remix theme={null}
// app/routes/api.$.ts`
import type { LoaderFunctionArgs, ActionFunctionArgs } from "@remix-run/node"
import { nile } from "~/nile"; // the nile instance
export const loader = async ({ request }: LoaderFunctionArgs) => {
const method = request.method.toUpperCase();
return nile.handlers[method](request)
};
export const action = async ({ request }: ActionFunctionArgs) => {
const method = request.method.toUpperCase();
return nile.handlers[method](request)
};
```
```ts express theme={null}
// server.ts
import { Nile } from "@niledatabase/server";
import { express } from "@niledatabase/express";
const nile = Nile({ extensions: [express(app)] });
```
```ts nitro theme={null}
import { nile } from "~/nile"; // the nile instance
import { convertToRequest } from '@niledatabase/nitro'
export default defineEventHandler((event) => {
return convertToRequest(event, nile);
});
```
```ts JS theme={null}
async function startServer(req, res) {
try {
const method = req.method?.toUpperCase() as 'GET' | 'POST' | 'PUT' | 'DELETE';
if (!method || !nile.handlers[method]) {
res.writeHead(405, { 'Content-Type': 'text/plain' });
return res.end('Method Not Allowed');
}
const url = `http://${req.headers.host}${req.url}`;
const bodyChunks: Uint8Array[] = [];
req.on('data', chunk => bodyChunks.push(chunk));
req.on('end', async () => {
const body = Buffer.concat(bodyChunks);
const request = new Request(url, {
method,
headers: req.headers as HeadersInit,
body: method === 'POST' || method === 'PUT' ? body : undefined,
});
let response;
try {
response = await nile.handlers[method](request);
} catch (err) {
console.error(err);
res.writeHead(500, { 'Content-Type': 'text/plain' });
return res.end('Internal Server Error');
}
res.writeHead(response.status, Object.fromEntries(response.headers.entries()));
const respBody = await response.text();
res.end(respBody);
});
} catch (err) {
console.error(err);
res.writeHead(500, { 'Content-Type': 'text/plain' });
res.end('Unexpected Server Error');
}
};
```
```ts nile-client.ts theme={null}
import { getSession, signUp } from '@niledatabase/client'
```
With that, you've successfully set up nile-auth in your application.
Continue on to [Social providers](../singlesignon/discord), or check out how to [integrate into specific frameworks](/auth/frameworks/nextjs)
# Community
Source: https://thenile.dev/docs/auth/help/community
Join the Nile Auth community
We are so glad you're here and interested in contributing to Nile Auth!
Whether you are interested in reporting bugs, asking questions, answering questions, suggesting features,
writing blogs, recording tutorials or contributing code - we appreciate your help in making Nile Auth better.
Here are some ways you can get involved:
## Discord Community
Join our [Discord server](https://discord.com/invite/8UuBB84tTy) to:
* Get help from the community
* Share your experiences and opinions
* Connect with other developers
* Stay updated on new features
## GitHub Discussions
Please use our [GitHub discussion board](https://github.com/orgs/niledatabase/discussions) to:
* Report issues
* Submit feature requests
* Vote on issues and feature requests
* Find answers to your questions
* Find issues and feature requests to contribute to
We wrote a detailed guide on [how to report issues](/auth/contributing/report) - we suggest reviewing it. It will help us help you faster.
## Social Media
Follow us on:
* [Twitter/X](https://x.com/niledatabase)
* [LinkedIn](https://www.linkedin.com/company/niledatabase)
## Contributing Code
Want NileAuth to support a new provider? Need a new feature? Want to fix a bug?
We wrote a detailed guide on [how to contribute code](/auth/contributing/develop) to get you started.
And of course, feel free to reach out to us on [Discord](https://discord.com/invite/8UuBB84tTy) to discuss your ideas.
## Contributing Blogs, Examples and Tutorials
Did you write a blog, example or tutorial about Nile Auth? We would love to feature it on our website and social media.
Please reach out to us on [Discord](https://discord.com/invite/8UuBB84tTy).
## Support
Need additional help?
* [Read the FAQ](/auth/help/faq)
* [Error Reference](/auth/help/errors)
* [Contact Support](https://www.thenile.dev/contact-us)
# Errors
Source: https://thenile.dev/docs/auth/help/errors
Common errors and their solutions in Nile Auth
# Error Reference
A comprehensive guide to common errors in Nile Auth and how to resolve them.
## Authentication Errors
Authentication errors are errors that occur when a user attempts to authenticate with Nile Auth and something goes wrong.
In order to get the most information about the error, we recommend:
**Implement error handlers when using Nile Auth components**:
All errors caused by user input are returned as `200`, so you need to handle them in the `onSuccess` callback by checking if `res.ok` is `false`.
The error itself will be in the `res.text()` property. You can choose to display the error to the user, or silently handle it and log to the console.
Check the [component guides](/auth/components/) for examples.
**Enable debug mode when instantiating Nile Server SDK**:
This will log everything that happens in the Nile Auth SDK, including detailed error messages.
```js theme={null}
const nile = new Nile({
debug: true,
});
```
Without debug logging, authentication errors will show as `400` errors in your server logs.
### Invalid Credentials
This typically happens when the user has entered an incorrect email or password.
Use the error handler callback to display an error message to the user and implement a [password reset flow](/auth/email/password).
### User already exists
This happens when the user tries to sign up with an email that is already in use. They may have forgotten that they already have an account.
Use the error handler callback to display an error message to the user and direct them to sign in or reset their password.
### Invalid email
This happens when the user tries to sign up with an email that is not valid.
Use the error handler callback to display an error message to the user and direct them to sign up with a valid email.
## Configuration Errors
### Misconfigured environment variables
If you are missing any of the environment variables, you will get an error message in your server logs:
```
Error: User is required. Set NILEDB_USER as an environment variable or set `user` in the config options
```
Make sure you have set all the required environment variables:
```
NILEDB_USER=
NILEDB_PASSWORD=
NILEDB_API_URL=
NILEDB_POSTGRES_URL=
```
Review the [quickstart guide](/auth/quickstarts) for more information on how to set them up.
### Missing Social Provider Configuration
If you are missing the social provider configuration, but trying to use a social provider to sign in,
you will be redirected to a fallback page with plain email and password sign in.
In order to use a social provider, you need to configure it in the [Nile Console](https://console.thenile.dev).
You can read more about the social providers in the [single sign-on guide](/auth/singlesignon/google).
### Invalid Social Provider Configuration
If you have incorrectly configured the social provider configuration, the user will see an error message from the provider
which has some details about the error.
Review the error, the provider configuration in Nile Console and the configuration in the provider's developer console.
Typically the issue is that the redirect URL in the SignOn component does not match the redirect URL in the provider's developer console.
## Getting Help
* [Join our Discord Community](/auth/help/community)
* [Check our FAQ](/auth/help/faq)
* [Contact Support](https://www.thenile.dev/contact-us)
# FAQ
Source: https://thenile.dev/docs/auth/help/faq
Common questions and answers about Nile Auth
## General Questions
### What is Nile Auth?
Nile Auth is a multi-tenant authentication service that provides authentication and authorization features for multi-tenant applications.
This service ensures secure, scalable authentication while maintaining tenant isolation.
All B2B applications are multi-tenanted, and they all have concepts of tenants, users and data. Nile Auth brings these together
in a single service - authenticating users, managing tenants and access to their data.
### How is Nile Auth different from other auth providers?
* Nile Auth is purpose-built for multi-tenanted applications. It provides per-tenant authentication, authorization and user management.
* Nile Auth is an open source service - you can self-host it or use our hosted service
* Both managed and self-hosted options store all user data in your own database.
* Nile Auth supports a wide range of frameworks
* Nile Auth has beautiful and flexible UI components
## Technical Questions
### How does tenant isolation work?
The short answer is that Nile separates tenant's data by isolating each tenant into a virtual database - separate blocks in memory and on disk.
By default, all tenants share the same compute resources (CPU, memory, etc), but you can also provision dedicated compute resources for a subset of tenants.
When a user authenticates, Nile Auth is aware of the tenants the user belongs to and can enforce access to data based on the
user's tenant membership. Nile also handles the routing of each user's queries to the correct tenant's database.
This is just at the high level. You can read more about it in the [tenant isolation](/tenant-virtualization/tenant-isolation) documentation.
### Can I use Nile Auth with my existing database?
Yes, Nile Auth can be configured to use your existing database.
### How do I implement SSO?
SSO is not yet supported, but it is planned for the future.
## Security
### How does Nile Auth handle security?
Nile Auth uses industry standard security best practices. All communication is encrypted using TLS, all stored data is encrypted at rest.
Nile Auth enforces secure, http-only cookies to prevent cross-site scripting attacks. It uses CSRF tokens to prevent cross-site request forgery.
Whenever possible, Nile Auth sessions are stored in the database, where they can be inspected and revoked if needed.
You can read more about our use of [cookies](/auth/concepts/cookies), [jwt](/auth/concepts/jwt), [oauth](/auth/concepts/oauth) and [sessions](/auth/concepts/sessions).
At the database layer, security is enforced by Nile's virtual tenant databases and the isolation they provide.
## Pricing and Plans
### Is there a free tier?
Yes, we offer a generous free tier - with unlimited databases and users. You can see all the details on our [pricing page](https://www.thenile.dev/pricing).
### How does Nile Auth compare to other auth providers?
We literally give away authentication for free - you have unlimited tenants and unlimited users available in every pricing tier - including the free tier.
### What are the usage limits?
There are none. Everything is unlimited - tenants, users, databases.
### Which frameworks are supported?
We are adding new frameworks fast. For starters, we support Next.js, Express, React, Remix, and more.
Check the [frameworks](/auth/frameworks/express) pages for the latest information.
### Can I self-host Nile Auth?
Yes, you can self-host Nile Auth. We provide a docker image and a helm chart.
And of course, it is open source, so you can build and deploy it in any way you want.
### Where can I get help?
* [Error Reference](/auth/help/errors)
* [Join our Community](/auth/help/community)
* [Contact Support](https://www.thenile.dev/contact-us)
# Introduction
Source: https://thenile.dev/docs/auth/introduction
[Nile Auth](https://github.com/niledatabase/nile-auth) is a comprehensive B2B auth solution explicitly designed for multi-tenant applications. Nile Auth is fully open source and built on top of Nile’s Postgres. It allows you to store user and customer data in your Postgres database, giving you complete control over your information. You can choose to self-host Nile Auth or utilize our cloud version.
Integrating auth into your B2B application on the front and back end takes just a few minutes with Nile Auth’s B2B authentication features and customizable UI components. If you choose the hosted version, you can enjoy unlimited active users for authentication at no additional cost; you only pay for the database.
## Why Nile Auth?
### Purpose-built for multi-tenant apps
Nile re-engineers Postgres to make it easy to build and scale B2B apps. Our Auth product is also designed from the ground up to support multi-tenant applications on top of Nile's Postgres.
Auth solutions usually don’t support B2B apps, or the focus is not full-time. In Nile’s Auth case, we have designed the product for only B2B apps. Nile Auth will support the entire tenant lifecycle, including managing tenants, inviting users to tenants, overriding tenant-specific settings, tenant domains, and more.
Authenticated users can access data from any tenant they have access to - this access control is enforced at all layers - from the browser to the authentication service to the database itself. All authentication features can be enabled at the application level or disabled for a specific tenant.
### Unlimited active users
One of our key focuses in developing Nile Auth was to offer the ability to support unlimited active users. Traditionally, authentication providers set a fixed limit on the number of active users, often requiring additional payments as you scale up. However, with Nile Auth, there are no extra charges for active users. You will only pay for the PostgreSQL database, allowing you to store and scale to an unlimited number of active users without incurring additional costs.
### Customer and user data stored in your DB
One of the challenges we have faced with third-party auth providers is that user and customer data are locked in behind third-party APIs. There are a few issues with this approach
* Referencing and joining user data with other tables in your database. It gets hard to refer to the user data using foreign key constraints or SQL joins to query across user and other business tables.
* The synchronization process is async and poses consistency challenges. You could synchronize using a webhook or capture events into a changelog and apply them to the primary DB. Both approaches eventually create consistency problems. If other tables reference your application's customer or user data, users will face weird delays or could even lose data.
While Nile Postgres will integrate with third-party auth providers as well, we believe the ideal approach is to just have the Auth solution store the data in the primary DB.
There are also other benefits of building Nile Auth on Nile’s Postgres. Tenant management is not just an Auth problem. It is a data problem. Nile’s Postgres has taken a first principles approach to solving multi-tenant problems by Isolating tenant data, addressing noisy neighbor problems, providing usage and cost insights by tenants, placing tenants on different compute types and regions, and supporting all the DB operations at the tenant level. Using Nile Auth and Nile’s Postgres provides a truly end to end solution to building and scaling B2B apps.
### Comprehensive B2B auth features
Access a full suite of authentication features to secure your application thoroughly. Here are some of the features you get out of the box:
* Organization management
* User profiles
* Dashboard for managing users and organizations
* Tenant over-rides - manage authentication for each tenant individually.
* Multi-framework support - NextJS, Express, React, etc.
* Wide range of authentication methods - email/password, social login, magic link, etc.
* UI components for embedding in your application - simple, beautiful, and flexible
* Cookie-Based Authentication: Secure session management using HTTP-only cookies.
* JWT and Session Support: Uses cookies to maintain user sessions and optionally issues JWTs for client-side validation.
* Single Sign-On (SSO) Support: Optional integration with external identity providers.
Auth services really have to be comprehensive. Unlike other parts of the app where mixing and matching libraries and services is normal, mixing and matching two auth solutions is very challenging from a security POV - you need to keep them in sync, you need to make sure you correctly handle responses from two services, juggle multiple cookies and tokens (possibly with different expirations). Much safer to find one auth service that does everything you need and use that.
There is more on the roadmap, and our goal is to support a comprehensive list of features for B2B apps. We would [love to know](https://github.com/orgs/niledatabase/discussions?discussions_q=is%3Aopen+label%3Aauth) if you need any specific features not currently available. We also welcome community contributions.
### **Self-host or let Nile manage it**
One of our design goals was to make it easy to self-host Nile Auth. We believe we have made it easy for developers to use our managed solution or self-host the auth service. The auth service will still use hosted Nile’s Postgres, but one can get the benefit of running core security logic within their account in the cloud. For development purposes, the entire Nile stack, including Postgres and Nile Auth, is available as a [docker image](http://localhost:3003/getting-started/postgres_testing). Developers can test locally and use the hosted offering when deploying to production.
### Drop-in fully customizable auth UI modules
Easily integrate pre-built authentication UI modules into your application in five minutes. Add support for Google, GitHub, and more and override per-tenant. Nile’s open-source SDK includes beautiful and flexible React components that can be embedded in your application and customized to your liking. This includes the signup, login, organization switcher, user profile, social login buttons, and more.
## Design principles of Nile Auth
### Support multiple languages
To truly democratize B2B authentication, we wanted to build a solution that can be leveraged with as many languages as possible.
We currently support the Typescript/Javascript ecosystem but plan to support more languages.
We have published our public [Auth APIs](/auth/api-reference/auth/sign-in-to-the-application) and hope to provide more language coverage. We would also love the community to build and contribute SDKs for their favorite language.
### Auth as a service vs a library
Auth solutions have been tackled as a service and as a library. Based on our experience, a service-first approach is the most secure solution for B2B Authentication for several reasons.
1. Libraries are convenient till the new CISO in a B2B company mandates that auth has to be moved to a service for security reasons. Having the auth logic across many services is usually not what CISOs want. Security teams prefer to control the authentication logic centrally.
2. Security hotfixes are a nightmare with thick clients. When a security hole is identified, it becomes critical for B2B companies to deploy the fix immediately. It is much easier to hotfix the central service vs upgrading the library in multiple services.
3. In most cases, B2B Companies need to support multiple languages from the start. For example, companies build their data plane with Typescript but their control plane using Go. Users will authenticate against both the control plane and the data plane.
We have invested in making it easy to launch and operate the Nile Auth service.
### Server side authentication
Server-side authentication is a core design principle of Nile Auth, ensuring flexibility, security, and broad applicability. By focusing on server-side APIs and auto-generated routes, developers can build their own UI without being locked into a specific framework. This approach also enables authentication for API-only services, ensuring secure endpoint protection. Additionally, Nile Auth is built on strong security primitives, including Secure, HttpOnly cookies to prevent XSS, CSRF protection, and session-backed authentication instead of relying solely on JWTs. This aligns with best security practices, addressing concerns like those raised by CISOs who prioritize moving authentication logic to the backend for consistency and security across web, API, and CLI environment
### Open source
We have spent years doing open-source and believe this is the right path for Nile Auth. Open source will help build trust and make Nile Auth extremely secure. It raises the bar to build a world-class service and enables the community to contribute. We want to build a developer-first community around Nile Auth and make it easy to contribute. You can look at our [contributing guidelines](https://github.com/niledatabase/nile-auth/blob/main/CONTRIBUTING.md) if you are interested in adding features or fixing issues.
## Next Steps
* [Quickstart Guide](/auth/quickstarts)
* [Self Hosting](/auth/selfhosting)
* [Concepts](/auth/concepts/tenants)
# Magic Link
Source: https://thenile.dev/docs/auth/magiclink
Magic Link Authentication
Magic Link authentication provides a passwordless sign-in experience where users
receive a secure link via email to authenticate. This method eliminates the need for
passwords while maintaining security.
## Overview
Magic Link authentication flow:
1. User enters their email address
2. They receive a secure link with an authentication token via email. The token is valid for 4 hours by default and is saved in the `auth.verification_tokens` table in your database.
3. Clicking the link automatically exchanges the token for a session, logs them into your application and redirects them to the `callbackUrl` configured in the `sign` component.
## Implementation Steps
1. If you haven't signed up for Nile yet, [sign up here](https://console.thenile.dev) and follow the steps to create a database.
2. Navigate to **Database Settings** in your database's UI at [console.thenile.dev](https://console.thenile.dev).
3. Go to **Connection** settings.
4. Select the CLI icon, and click **Generate credentials**
5. **Copy** the required credentials and **store them in an `.env` file** so they can be used in the application to connect to the Nile auth service.
```bash .env theme={null}
NILEDB_USER=niledb_user
NILEDB_PASSWORD=niledb_password
NILEDB_API_URL=https://us-west-2.api.thenile.dev/v2/databases/
NILEDB_POSTGRES_URL=postgres://us-west-2.db.thenile.dev:5432/
```
Ensure you have configured your email settings in the Nile Dashboard. You'll need a valid
[SMTP provider](/auth/email/customsmtp) configured to send emails from your
application, and [email templates](/auth/email/templates) configured for the magic link.
Click on Providers tab under Configure. Under Email, enable passwordless login.
This guide uses Next.js with App Router, Typescript and Tailwind CSS. If you have a different framework in mind, you can find additional guides under "Frameworks"
in the sidebar. Initialize a new Next.js project with the following command and give it a name:
```bash theme={null}
npx create-next-app@latest nile-app --yes
```
```bash theme={null}
npm install @niledatabase/server @niledatabase/react
@niledatabase/client @niledatabase/nextjs
```
Your application must expose API routes to handle authentication operations.
Create a folder called `api` under the `app` folder and a folder called `[...nile]` under it:
```bash theme={null}
mkdir -p app/api/\[...nile\]
```
Create following files handle the calls to your server, as well as expose the `nile` instance to your application:
```typescript app/api/[...nile]/nile.ts theme={null}
import { Nile } from '@niledatabase/server';
import { nextJs } from '@niledatabase/nextjs';
export const nile = Nile({ extensions: [nextJs] });
```
```typescript app/api/[...nile]/route.ts theme={null}
import { nile } from './nile';
export const { POST, GET, DELETE, PUT } = nile.handlers;
```
The ` ` component is used to send a magic link to the user's email address.
```jsx theme={null}
import { EmailSignIn } from "@niledatabase/react";
export default function SignUpPage() {
return
}
```
# From Auth0
Source: https://thenile.dev/docs/auth/migration/fromauth0
Learn how to migrate from Auth0 to Nile Auth
If you are currently using Auth0, this guide will help you migrate to Nile Auth.
Migrating production applications from one auth provider to another is a non-trivial task,
and we recommend you do thorough testing.
Before you begin, you'll need a Nile account. If you don't have one, you can sign up [here](https://console.thenile.dev).
You should also review the [Nile Auth](/auth/introduction) documentation and validate that Nile Auth meets
your application requirements (That we support the providers you need, for instance). If you notice any features missing, [please let us know!](https://github.com/orgs/niledatabase/discussions?discussions_q=label%3Aauth)
## Migration Steps
This step is necessary if you use Auth0's email/password or passwordless login methods.
In this case, your users' data is stored in Auth0's database, and you will need to export it.
Auth0 has a [friendly UI for exporting user data](https://auth0.com/docs/manage-users/user-migration/user-import-export-extension), but unfortunately, it does not contain the organizations that users belong to.
Therefore, you'll need to use the management API to first list all the organizations, and then for each organization,
list all the users.
Here is an example of how to do this:
```typescript [expandable] theme={null}
var myHeaders = new Headers();
myHeaders.append('Accept', 'application/json');
var requestOptions = {
method: 'GET',
headers: myHeaders,
redirect: 'follow',
};
const allUsers = [];
const response = await fetch(
'https://login.auth0.com/api/v2/organizations',
requestOptions,
);
const organizations = await response.json();
// Write organizations to a file
fs.writeFileSync(
'auth0-organizations.json',
JSON.stringify(organizations, null, 2),
);
for (const organization of organizations) {
const response = await fetch(
`https://login.auth0.com/api/v2/organizations/${organization.id}/users`,
requestOptions,
);
const users = await response.json();
// Add organization name to each user and collect them
users.forEach((user) => {
allUsers.push({
...user,
organizationName: organization.name,
});
});
}
// Write all users to a file
const fs = require('fs');
fs.writeFileSync('auth0-users.json', JSON.stringify(allUsers, null, 2));
```
Note that Auth0 does not export user's password hashes through these tools, you will need to open a support ticket to get
this data.
You'll need a small script to import the user data into Nile Auth.
The script will iterate over the exported user data and create users in Nile Auth.
The following example script shows how to import user data from a JSON file:
```typescript [expandable] theme={null}
import { Nile } from '@niledatabase/server';
import * as fs from 'fs';
interface Auth0User {
Id: string;
Nickname: string;
Name: string;
email: string;
Connection: string;
'Created At': string;
'Updated At': string;
}
interface Auth0Organization {
id: string;
name: string;
display_name: string;
metadata?: Record;
}
async function migrateUsers(nile: any) {
try {
// Read and parse the JSON file
const rawData = fs.readFileSync('auth0-users.json', 'utf8');
const users: Auth0User[] = JSON.parse(rawData);
let newUser;
// Process each user
for (const user of users) {
try {
// Generate a temporary password (you might want to adjust this strategy)
const tempPassword = `temp-${Math.random().toString(36).slice(-8)}`;
// Get the tenant id from the organization name
const tenant = await nile.db.query(
`SELECT id FROM tenants WHERE name = $1`,
[user.organizationName],
);
if (!tenant.rows[0]) {
console.error(`Tenant ${user.organizationName} not found`);
continue;
}
nile.tenant_id = tenant.rows[0].id; // Set the tenant id so the user is created in the correct tenant
const preferredName = user.Nickname || user.Name;
const {
rows: [newUser],
} = await nile.db.query(
'insert into users.users (email, password, preferredName) values ($1, $2, $3) returning *',
[user.email, tempPassword, preferredName],
);
// uncomment this to update the password hash in the credentials table
// only do this if you exported the password hashes in bcrypt format
// await nile.db.query(
// `UPDATE auth.credentials SET payload = jsonb_build_object('crypt', 'crypt-bf/8', 'hash', $1)
// WHERE user_id = $2 AND method = 'PASSWORD';`,
// [user.Password, newUser.Id]
// )
console.log(`Successfully migrated user: ${user.email}`);
// You might want to store the temporary passwords to communicate them to users
console.log(`Temporary password for ${user.email}: ${tempPassword}`);
} catch (error) {
console.error(`Failed to migrate user ${user.email}:`, error);
// Continue with next user even if one fails
continue;
}
}
console.log('Migration completed');
} catch (error) {
console.error('Migration failed:', error);
}
}
// This is necessary only if the organizations don't already exist in Nile
async function migrateTenants(nile: any) {
try {
// Read and parse the organizations JSON file
const rawData = fs.readFileSync('auth0-organizations.json', 'utf8');
const organizations: Auth0Organization[] = JSON.parse(rawData);
// Process each organization
for (const org of organizations) {
try {
const tenant = await nile.tenants.create({
name: org.name,
});
console.log(`Successfully created tenant: ${org.name}`);
} catch (error) {
console.error(`Failed to create tenant ${org.name}:`, error);
// Continue with next organization even if one fails
continue;
}
}
console.log('Tenant migration completed');
} catch (error) {
console.error('Tenant migration failed:', error);
}
}
// Single initialization of Nile
async function migrateAll() {
const nile = Nile();
await migrateTenants(nile);
await migrateUsers(nile);
}
migrateAll();
```
If you did not have the password hashes exported, your users will need to reset their passwords after the migration (unless you used passwordless authentication) - you can read how to do this in our [password reset docs](/auth/email/password).
For staged migration, or for edge cases, you can implement the fallback strategy in your application. Attempt authentication with
Nile Auth, and in the failure handler, attempt the Auth0 authentication. If Auth0 authentication succeeds,
you can insert into the `users.users` table.
If you used social identity providers with , you will need to configure the same providers in Nile Auth.
You can configure social providers by going to "Tenants and Users" page in [Nile Console](https://console.thenile.dev)
and selecting the "Providers" tab.
Simply click on the provider you want to configure and you'll see the provider configuration page.
All providers use OAuth, so you'll need to provide the OAuth client ID and secret. It is recommended to use the same client
ID and secret as the one used in . And to use the same redirect URI in the social provider configuration.
You can see more details on how to configure and use each provider in the [Single Sign-On](/auth/singlesignon/google) section of the documentation.
## Testing
We recommend you do thorough testing in a staging environment before migrating to production.
Before starting a migration, both in staging and in production, we recommend you create a few test users and organizations in your existing auth provider.
Make sure you have a mix of email/password, passwordless and social providers users.
Then, after migrating to Nile Auth, test the following (both in test and in production):
* Sign up and sign in with email/password, passwordless and social providers.
* Sign in with email/password, passwordless and social providers.
* Sign out
* Reset password
* Email verification
* Validate that the users have access to the correct organizations.
It is also important to test **negative** flows. For example, trying to sign in with an invalid password,
or an invalid email/password combination. You want to be sure that unauthorized users are not able to sign in.
Finally, use the ["Tenant and Users" page in Nile Console](/auth/dashboard/managing-tenants) to verify that the all the expected user data and
metadata was migrated, and that the users have access to the correct organizations.
## Best Practices
Here are some recommended practices for a smooth migration from Auth0 to Nile Auth:
1. **Staged Migration**: Consider migrating users in batches rather than all at once.
Start with a small subset of non-critical users or test accounts. Monitor for issues and gather feedback before proceeding with larger groups.
This will allow you to test the migration and make sure it works as expected before migrating all users. The "fallback strategy"
described in the migration steps can be used to support both Auth0 and Nile Auth during the migration.
2. **Communication Plan**: Notify users well in advance of the migration. Provide clear instructions for any actions they n
eed to take (like password resets). Set up support channels for users who encounter issues during the transition.
3. **Backup and Rollback Plan**: Keep backups of all Auth0 data before starting the migration.
Maintain the Auth0 configuration until the migration is complete and verified.
Have a clear rollback plan in case of critical issues.
4. **Monitoring and Validation**: Set up monitoring for authentication failures during and after migration.
Implement logging to track successful and failed migrations. Validate user counts and access permissions after migration.
## Related Topics
* [Firebase Migration](/auth/migration/fromfirebase)
* [User Management](/auth/concepts/users)
* [Tenant Management](/auth/concepts/tenants)
# From Firebase
Source: https://thenile.dev/docs/auth/migration/fromfirebase
Learn how to migrate from Firebase Authentication to Nile Auth
If you are currently using Firebase Authentication, this guide will help you migrate to Nile Auth.
Migrating production applications from one auth provider to another is a non-trivial task,
and we recommend you do thorough testing.
Before you begin, you'll need a Nile account. If you don't have one, you can sign up [here](https://console.thenile.dev).
## Migration Steps
This step is necessary if you use Firebase's email/password or passwordless login methods.
In this case, your users' data is stored in Firebase, and you will need to export it.
Firebase has a [friendly CLI tool for exporting user data](https://firebase.google.com/docs/cli/auth#auth-export), but unfortunately, it does not contain the organizations that users belong to.
Therefore, we need to use the Firebase Admin SDK to first list all the organizations, and then for each organization, list all the users.
This requires installing and configuring the Firebase Admin SDK. Follow the [Firebase Admin SDK setup guide](https://cloud.google.com/identity-platform/docs/install-admin-sdk) for more details.
Here is an example script that exports the tenants and users:
```typescript [expandable] theme={null}
import * as admin from 'firebase-admin';
import * as fs from 'fs';
async function listAllTenants(nextPageToken, allTenants = []) {
const result = await admin
.auth()
.tenantManager()
.listTenants(100, nextPageToken);
allTenants.push(...result.tenants.map((tenant) => tenant.toJSON()));
if (result.pageToken) {
return await listAllTenants(result.pageToken, allTenants);
}
return allTenants;
}
async function listAllUsers(tenantId, nextPageToken, allUsers = []) {
const tenantAuth = admin.auth().tenantManager().authForTenant(tenantId);
try {
const listUsersResult = await tenantAuth.listUsers(1000, nextPageToken);
// Add users with their tenant ID
allUsers.push(
...listUsersResult.users.map((user) => ({
...user.toJSON(),
tenantId: tenantId,
})),
);
if (listUsersResult.pageToken) {
return await listAllUsers(tenantId, listUsersResult.pageToken, allUsers);
}
return allUsers;
} catch (error) {
console.error(`Error listing users for tenant ${tenantId}:`, error);
return allUsers;
}
}
async function exportAllTenantsAndUsers() {
try {
// Get all tenants first
const tenants = await listAllTenants();
const allUsers = [];
// For each tenant, get all users
for (const tenant of tenants) {
console.log(`Fetching users for tenant: ${tenant.tenantId}`);
const tenantUsers = await listAllUsers(tenant.tenantId, null);
allUsers.push(...tenantUsers);
}
// Write tenants and users to separate files
fs.writeFileSync('firebase-tenants.json', JSON.stringify(tenants, null, 2));
fs.writeFileSync('firebase-users.json', JSON.stringify(allUsers, null, 2));
console.log(
'Export completed! Data written to firebase-tenants.json and firebase-users.json',
);
} catch (error) {
console.error('Export failed:', error);
}
}
// Start the export
exportAllTenantsAndUsers();
```
Firebase supports exporting password hashes, but note that you can only load them into Nile Auth if you are using **bcrypt**.
To determine the password hash parameters used for your project navigate to the `Authentication > Users` section of the Firebase console and click the three dots icon above the list of users.
You'll need a small script to import the user data into Nile Auth.
The script will iterate over the exported user data and create users in Nile Auth.
The following example script shows how to import user data from a JSON file:
```typescript [expandable] theme={null}
import { Nile } from '@niledatabase/server';
import * as fs from 'fs';
interface FirebaseUser {
tenantId: string;
uid: string;
email: string;
emailVerified: boolean;
disabled: boolean;
metadata: {
lastSignInTime: string;
lastSignInIp: string;
lastLoginAt: string;
lastLoginIp: string;
};
password: string;
'Updated At': string;
}
interface FirebaseOrganization {
id: string;
name: string;
display_name: string;
metadata?: Record;
}
async function migrateUsers(nile: any) {
try {
// Read and parse the JSON file
const rawData = fs.readFileSync('firebase-users.json', 'utf8');
const users: FirebaseUser[] = JSON.parse(rawData);
let newUser;
// Process each user
for (const user of users) {
try {
// Generate a temporary password (you might want to adjust this strategy)
const tempPassword = `temp-${Math.random().toString(36).slice(-8)}`;
// Get the tenant id from the organization name
const tenant = await nile.db.query(
`SELECT id FROM tenants WHERE name = $1`,
[user.tenantId],
);
if (!tenant.rows[0]) {
console.error(`Tenant ${user.tenantId} not found`);
continue;
}
nile.tenant_id = tenant.rows[0].id; // Set the tenant id so the user is created in the correct tenant
const preferredName = user.Nickname || user.Name;
const {
rows: [newUser],
} = await nile.db.query(
'insert into users.users (email, password, preferredName) values ($1, $2, $3) returning *',
[user.email, tempPassword, preferredName],
);
// uncomment this to update the password hash in the credentials table
// only do this if you exported the password hashes in bcrypt format
// await nile.db.query(
// `UPDATE auth.credentials SET payload = jsonb_build_object('crypt', 'crypt-bf/8', 'hash', $1)
// WHERE user_id = $2 AND method = 'PASSWORD';`,
// [user.Password, newUser.Id]
// )
console.log(`Successfully migrated user: ${user.email}`);
// You might want to store the temporary passwords to communicate them to users
console.log(`Temporary password for ${user.email}: ${tempPassword}`);
} catch (error) {
console.error(`Failed to migrate user ${user.email}:`, error);
// Continue with next user even if one fails
continue;
}
}
console.log('Migration completed');
} catch (error) {
console.error('Migration failed:', error);
}
}
// This is necessary only if the organizations don't already exist in Nile
async function migrateTenants(nile: any) {
try {
// Read and parse the organizations JSON file
const rawData = fs.readFileSync('firebase-tenants.json', 'utf8');
const organizations: FirebaseOrganization[] = JSON.parse(rawData);
// Process each organization
for (const org of organizations) {
try {
const tenant = await nile.tenants.create({
name: org.name,
});
console.log(`Successfully created tenant: ${org.name}`);
} catch (error) {
console.error(`Failed to create tenant ${org.name}:`, error);
// Continue with next organization even if one fails
continue;
}
}
console.log('Tenant migration completed');
} catch (error) {
console.error('Tenant migration failed:', error);
}
}
// Single initialization of Nile
async function migrateAll() {
const nile = Nile({});
await migrateTenants(nile);
await migrateUsers(nile);
}
migrateAll();
```
If you did not have the password hashes imported to Nile, your users will need to reset their passwords after the migration (unless you used passwordless authentication) - you can read how to do this in our [password reset docs](/auth/email/password).
For staged migration, or for edge cases, you can implement the fallback strategy in your application. Attempt authentication with
Nile Auth, and in the failure handler, attempt the Auth0 authentication. If Auth0 authentication succeeds,
you can insert into `users.users` to create the user.
If you used social identity providers with , you will need to configure the same providers in Nile Auth.
You can configure social providers by going to "Tenants and Users" page in [Nile Console](https://console.thenile.dev)
and selecting the "Providers" tab.
Simply click on the provider you want to configure and you'll see the provider configuration page.
All providers use OAuth, so you'll need to provide the OAuth client ID and secret. It is recommended to use the same client
ID and secret as the one used in . And to use the same redirect URI in the social provider configuration.
You can see more details on how to configure and use each provider in the [Single Sign-On](/auth/singlesignon/google) section of the documentation.
## Testing
We recommend you do thorough testing in a staging environment before migrating to production.
Before starting a migration, both in staging and in production, we recommend you create a few test users and organizations in your existing auth provider.
Make sure you have a mix of email/password, passwordless and social providers users.
Then, after migrating to Nile Auth, test the following (both in test and in production):
* Sign up and sign in with email/password, passwordless and social providers.
* Sign in with email/password, passwordless and social providers.
* Sign out
* Reset password
* Email verification
* Validate that the users have access to the correct organizations.
It is also important to test **negative** flows. For example, trying to sign in with an invalid password,
or an invalid email/password combination. You want to be sure that unauthorized users are not able to sign in.
Finally, use the ["Tenant and Users" page in Nile Console](/auth/dashboard/managing-tenants) to verify that the all the expected user data and
metadata was migrated, and that the users have access to the correct organizations.
## Best Practices
Here are some recommended practices for a smooth migration from Auth0 to Nile Auth:
1. **Staged Migration**: Consider migrating users in batches rather than all at once.
Start with a small subset of non-critical users or test accounts. Monitor for issues and gather feedback before proceeding with larger groups.
This will allow you to test the migration and make sure it works as expected before migrating all users. The "fallback strategy"
described in the migration steps can be used to support both Auth0 and Nile Auth during the migration.
2. **Communication Plan**: Notify users well in advance of the migration. Provide clear instructions for any actions they n
eed to take (like password resets). Set up support channels for users who encounter issues during the transition.
3. **Backup and Rollback Plan**: Keep backups of all Auth0 data before starting the migration.
Maintain the Auth0 configuration until the migration is complete and verified.
Have a clear rollback plan in case of critical issues.
4. **Monitoring and Validation**: Set up monitoring for authentication failures during and after migration.
Implement logging to track successful and failed migrations. Validate user counts and access permissions after migration.
## Related Topics
* [Auth0 Migration](/auth/migration/fromauth0)
* [User Management](/auth/concepts/users)
* [Security Concepts](/auth/concepts/jwt)
# From NextAuth.js
Source: https://thenile.dev/docs/auth/migration/fromnextauth
Learn how to migrate from NextAuth.js to Nile Auth
If you are currently using NextAuth.js, this guide will help you migrate to Nile Auth.
Migrating production applications from one auth provider to another is a non-trivial task,
and we recommend you do thorough testing.
Before you begin, you'll need a Nile account. If you don't have one, you can sign up [here](https://console.thenile.dev).
You should also review the [Nile Auth](/auth/introduction) documentation and validate that Nile Auth meets
your application requirements (That we support the providers you need, for instance). If you notice any features missing, [please let us know!](https://github.com/orgs/niledatabase/discussions?discussions_q=label%3Aauth)
## Migration Steps
NextAuth.js is an authentication library, rather than a service.
This means that your users, sessions and accounts are stored in a database.
If your existing database is PostgreSQL, you can use Nile Auth with the same database and this document will guide you through the migration.
Otherwise, you'll need to migrate your data to PostgreSQL. Data migrations to PostgreSQL are not covered is beyond the scope of this guide,
but you can reach out to us for [help](auth/help/community).
You can see the schema of the auth tables in Nile Auth in our [built-in tables documentation](/auth/concepts/builtintables).
If you are using the standard Postgres adapter for NextAuth.js, your schema is documented [here](https://authjs.dev/getting-started/adapters/pg?framework=next-js). This schema is the same for NextAuth versions 3, 4 and 5 (also called Auth.js).
As you can see, there are quite a few similarities between the two schemas. Making the migration relatively straightforward.
First, if you are not using Nile, you'll need to create the tables as [documented](/auth/concepts/builtintables).
Assuming you have these tables, either built into Nile or created by yourself, you'll need to migrate the data from the NextAuth.js tables to the Nile Auth tables. In the example queries below, I'm assuming that the NextAuth.js tables are in the `nextauth` schema. Modify the queries below to match your schema.
If you don't mind users re-authenticating, you don't need to migrate the
`sessions` and `verification_tokens` tables.
**Users table**
Nile Auth uses `uuid` type for the `id` column, while NextAuth.js uses `serial` type (which is actually an integer).
Since there is no automatic conversion between these types, we recommend adding a new column to store the NextAuth.js user id,
allowing you to keep existing references to these IDs intact.
```sql theme={null}
ALTER TABLE auth.users ADD COLUMN nextauth_id integer;
```
To copy the data from the NextAuth.js users table to the Nile Auth users table, you can use the following query:
```sql theme={null}
insert into users.users (email, name, email_verified, image, nextauth_id)
select email, name, email_verified, image, id as nextauth_id
from nextauth.users;
```
**Sessions table**
The sessions table in Nile Auth is almost identical to the one in NextAuth.js.
We just need to convert the user IDs to be Nile Auth user IDs instead of NextAuth.js user IDs.
```sql theme={null}
insert into auth.sessions (session_token, user_id, expires_at)
select s.session_token, u.id as user_id, s.expires_at
from nextauth.sessions s
join users.users u on u.nextauth_id = s.user_id;
```
**Verification Tokens table**
The verification tokens table in Nile Auth is identical to the one in NextAuth.js. No changes are needed.
```sql theme={null}
insert into auth.verification_tokens (identifier, token, expires)
select identifier, token, expires
from nextauth.verification_tokens;
```
**Accounts / Credentials table**
Nile Auth uses the `auth.credentials` table to store account information instead of the `accounts` table.
The query below will migrate the data from the `accounts` table to the `credentials` table.
```sql theme={null}
insert into auth.credentials (
user_id,
method,
provider,
provider_account,
payload
)
select
u.id as user_id,
'oidc'::authentication_method as method, -- all NextAuth.js accounts are oidc
a.provider,
a."providerAccountId" as provider_account,
jsonb_build_object(
'refresh_token', a.refresh_token,
'access_token', a.access_token,
'expires_at', a.expires_at,
'id_token', a.id_token,
'scope', a.scope,
'session_state', a.session_state,
'token_type', a.token_type,
'type', a.type
) as payload
from nextauth.accounts a
join users.users u on u.nextauth_id = a."userId";
```
Nile Auth is a multi-tenant authentication service. By linking users to tenants,
you can control their access to resources in your application and use Nile Auth organization components in your application.
This section assumes that you are already using Nile as your database, your tenants exist in `tenants` table, and your data is in tenant-aware tables with tenant isolation.
Migrating data from existing database into Nile's virtual tenant databases is out of scope for this guide, but you can reach out to us for [help](auth/help/community).
In order to link users to tenants, you'll need to populate the `users.user_tenants` table with the relationships between users and tenants.
You'll need to convert the current mapping of NextAuth.js users to tenants to the new Nile Auth user\_tenants table.
For example, if you have a `memberships` table with this mapping, you can use the following query to migrate the data:
```sql theme={null}
insert into users.user_tenants (user_id, tenant_id)
select u.id as user_id, m.tenant_id as tenant_id
from memberships m
join users.users u on u.nextauth_id = m.user_id;
```
If you used social identity providers with , you will need to configure the same providers in Nile Auth.
You can configure social providers by going to "Tenants and Users" page in [Nile Console](https://console.thenile.dev)
and selecting the "Providers" tab.
Simply click on the provider you want to configure and you'll see the provider configuration page.
All providers use OAuth, so you'll need to provide the OAuth client ID and secret. It is recommended to use the same client
ID and secret as the one used in . And to use the same redirect URI in the social provider configuration.
You can see more details on how to configure and use each provider in the [Single Sign-On](/auth/singlesignon/google) section of the documentation.
If you used NextAuth.js passwordless authentication, you'll need to configure your SMTP server and email templates in Nile Auth.
Refer to the [email configuration](/auth/dashboard/configurations#email-configuration) guide to learn how to do this.
In addition to the data migration, you'll need to update your application to use Nile Auth.
Refer to the [NextJS](/auth/frameworks/nextjs) guide to learn how to set up Nile Auth environment variables and routes in your application.
These will replace the NextAuth.js routes in your application.
NileAuth routes are purely backend and do not serve any frontend content. This means that you'll need to create login/signup pages in your application with
Nile's customizable components. The NextJS guide above has examples of how to do this.
Note that unlike NextAuth.js, you don't configure social providers and passwordless authentication in the application
* you already configured them in the **Social Providers** and **Passwordless Authentication** steps above.
## Testing
We recommend you do thorough testing in a staging environment before migrating to production.
Before starting a migration, both in staging and in production, we recommend you create a few test users and organizations in your existing auth provider.
Make sure you have a mix of email/password, passwordless and social providers users.
Then, after migrating to Nile Auth, test the following (both in test and in production):
* Sign up and sign in with email/password, passwordless and social providers.
* Sign in with email/password, passwordless and social providers.
* Sign out
* Reset password
* Email verification
* Validate that the users have access to the correct organizations.
It is also important to test **negative** flows. For example, trying to sign in with an invalid password,
or an invalid email/password combination. You want to be sure that unauthorized users are not able to sign in.
Finally, use the ["Tenant and Users" page in Nile Console](/auth/dashboard/managing-tenants) to verify that the all the expected user data and
metadata was migrated, and that the users have access to the correct organizations.
## Best Practices
Here are some recommended practices for a smooth migration from Auth0 to Nile Auth:
1. **Staged Migration**: Consider migrating users in batches rather than all at once.
Start with a small subset of non-critical users or test accounts. Monitor for issues and gather feedback before proceeding with larger groups.
This will allow you to test the migration and make sure it works as expected before migrating all users. The "fallback strategy"
described in the migration steps can be used to support both Auth0 and Nile Auth during the migration.
2. **Communication Plan**: Notify users well in advance of the migration. Provide clear instructions for any actions they n
eed to take (like password resets). Set up support channels for users who encounter issues during the transition.
3. **Backup and Rollback Plan**: Keep backups of all Auth0 data before starting the migration.
Maintain the Auth0 configuration until the migration is complete and verified.
Have a clear rollback plan in case of critical issues.
4. **Monitoring and Validation**: Set up monitoring for authentication failures during and after migration.
Implement logging to track successful and failed migrations. Validate user counts and access permissions after migration.
## Related Guides
* [Firebase Migration](/auth/migration/fromfirebase)
* [Auth0 Migration](/auth/migration/fromauth0)
# Express.js
Source: https://thenile.dev/docs/auth/quickstarts/express
Integrate Nile Auth with Express.js applications
Learn how to integrate Nile Auth with your Express.js application. The integration allows the application to interact with Nile's APIs and databases,
providing tenant-aware data management.
This guide provides an overview of how to use Nile-Auth core functionality with an Express.js application. We'll cover the following topics:
* Authentication, cookies, sessions
* Tenant isolation
* Securing endpoints
It is important to note that the Auth service is designed to work with actions
a **user** would take in the context of your B2B web application.
## Authentication
```bash theme={null}
mkdir express-app
cd express-app
npm init -y
```
```bash theme={null}
npm install @niledatabase/server @niledatabase/express express dotenv --save
```
1. If you haven't signed up for Nile yet, [sign up here](https://console.thenile.dev) and follow the steps to create a database.
2. Navigate to **Database Settings** in your database's UI at [console.thenile.dev](https://console.thenile.dev).
3. Go to **Connection** settings.
4. Select the CLI icon, and click **Generate credentials**
5. **Copy** the required credentials and **store them in an `.env` file** so they can be used in the application to connect to the Nile auth service.
```bash .env theme={null}
NILEDB_USER=niledb_user
NILEDB_PASSWORD=niledb_password
NILEDB_API_URL=https://us-west-2.api.thenile.dev/v2/databases/
NILEDB_POSTGRES_URL=postgres://us-west-2.db.thenile.dev:5432/
```
Create a new file called `server.mjs` and add the following code:
```javascript server.mjs theme={null}
import 'dotenv/config';
import express from 'express';
import { Nile } from '@niledatabase/server';
import { express as nileExpress } from '@niledatabase/express';
const startServer = async () => {
try {
const app = express();
const nile = Nile({
extensions: [nileExpress(app)],
});
app.use(express.json());
app.use(express.urlencoded({ extended: true }));
const PORT = process.env.PORT || 3040;
app.listen(PORT, () => {
console.log(`Server is running on port ${PORT}`);
});
} catch (error) {
console.error('Error starting server:', error);
process.exit(1);
}
};
startServer();
```
```bash theme={null}
node server.mjs
```
Nile auth uses cookies to store session information. To obtain them via cURL, create a file called `get_cookies.sh` and add the following code:
```bash get_cookies.sh theme={null}
#!/bin/bash
# Ensure EMAIL and PASSWORD are provided
if [ $# -lt 2 ]; then
echo "Usage: $0 [API_URL]"
exit 1
fi
EMAIL="$1"
PASSWORD="$2"
API_URL="${3:-http://localhost:3040}" # Default to localhost if not provided
# Define cookie file names
csrf_cookie_file="csrf_cookies.txt"
login_cookie_file="login_cookies.txt"
# Define API endpoints
CSRF_URL="$API_URL/api/auth/csrf"
LOGIN_URL="$API_URL/api/signup"
# Fetch CSRF token and store cookies
csrf_token=$(curl -s -X GET "$CSRF_URL" -c "$csrf_cookie_file" | jq -r '.csrfToken')
# Exit if CSRF token is missing
[ -z "$csrf_token" ] || [ "$csrf_token" == "null" ] && { echo "Failed to retrieve CSRF token"; exit 1; }
# Perform login request using CSRF token and cookies
curl -s -X POST "$LOGIN_URL" \
-H "Content-Type: application/x-www-form-urlencoded" \
-b "$csrf_cookie_file" \
--cookie-jar "$login_cookie_file" \
--data-urlencode "csrfToken=$csrf_token" \
--data-urlencode "email=$EMAIL" \
--data-urlencode "password=$PASSWORD" >/dev/null
# Output login cookie file info
echo "Login successful. Use $login_cookie_file for authenticated requests:"
echo "curl \"$API_URL/api/me\" -b $login_cookie_file"
```
Set the permissions to be executable
```bash theme={null}
chmod +x get_cookies.sh
```
Run the command with the required params
````bash theme={null}
./get_cookies.sh spongebob@squarepants.com no-this-is-patrick
You should see the following output
```bash
# Login successful. Use login_cookies.txt for authenticated requests:
# curl "http://localhost:3040/api/me" -b login_cookies.txt
````
You can then curl the API with the cookies
```bash theme={null}
curl "http://localhost:3040/api/me" -b login_cookies.txt
# You should see the following output
# {"email":"spongebob@squarepants.com","name":"Spongebob Squarepants","tenantId":"0192830912830912"}
```
## Tenant Isolation and Secure Endpoints
Since Nile-Auth is designed to work with B2B applications, it is important to understand how to work with tenants, their access to data,
and how to secure endpoints.
We are going to extend the previous example with new functionality.
We'll add a new table to the database, and a new endpoint that queries the data, making sure the
endpoint is both secure and isolated to the tenant.
You do not need a new endpoint in order to extand your application with tenant functionality. Nile's SDK includes generated routes for managing tenants.
We just need to call them:
```bash theme={null}
curl -X POST 'localhost:3040/api/tenants' \
-H 'Content-Type: application/json' \
-d '{"name":"my first customer"}' \
-b login_cookies.txt
```
There are multiple ways to pass the current tenant to the web app on each request.
You can pass it as a param, a header, or a cookie. In this example we'll pass it as a param.
Add the following code to your `server.mjs` file, just after the `app.use(express.urlencoded({ extended: true }));` line.
This will extract the tenant ID from the request params, and configure the nile client to use it as
the current tenant before handling any request.
`server.mjs`
```javascript theme={null}
app.param('tenantId', (req, res, next, tenantId) => {
nile.withContext({ tenantId });
next();
});
```
In Nile console or another database client of your choice, run the following SQL to create a new table called `todos`
and populate it with some example tasks.
```sql theme={null}
CREATE TABLE IF NOT EXISTS todos (
id uuid DEFAULT (gen_random_uuid()),
tenant_id uuid,
title varchar(256),
estimate varchar(256),
embedding vector(768),
complete boolean
);
insert into tenants (id, name) VALUES
('019637b6-6d72-774b-9796-8c35813f9f78', 'express_test_tenant');
-- Insert sample data. Make sure you replace the tenant_id with the one you created in the previous step.
INSERT INTO todos (tenant_id, title, estimate, complete)
VALUES
('019637b6-6d72-774b-9796-8c35813f9f78', 'Finish Express integration', '2h', false),
('019637b6-6d72-774b-9796-8c35813f9f78', 'Write documentation', '1h', false);
```
Add a route that takes a tenant Id and queries the database. if `app.param` is set
(as we did in the previous step), the query will automatically be [isolated](/tenant-virtualization/tenant-isolation) to the
current tenant. See how it returns data **only**for the tenant we requested even
if there are multiple tenants in the database and even though the query does not include a tenant\_id filter.
Add the following code to your `server.mjs` file, just after the `app.delete(paths.delete, handler);` line:
`server.mjs`
```javascript theme={null}
// Get all tasks for tenant
app.get('/api/tenants/:tenantId/todos', async (req, res) => {
try {
// the nile.tenantId is set in the previous `app.param`
const todos = await nile.query(`SELECT * FROM todos ORDER BY title`);
res.json(todos.rows);
return;
} catch (error) {
console.log('error listing tasks: ' + error.message);
res.status(500).json({ message: 'Internal Server Error' });
return;
}
});
```
The route we created is isolated to a specific tenant, however at this point, any user can call it. It is not secure.
Lets protect it by checking if the user is authenticated. Add the following code to your `server.mjs` file, just after the `app.get("/api/tenants/:tenantId/todos", async (req, res) => {` line:
```javascript theme={null}
const session = await nile.auth.getSession(req);
if (!session?.user) {
res.status(401).json({ message: 'Unauthorized' });
return;
}
```
You can add this logic in a middleware function, so it will be applied to all routes that need to be protected.
If you haven't already, run the server:
```bash theme={null}
node server.mjs
```
First, lets try to access the route without authentication. Make sure you replace the tenantId with the one you created in the previous step.
```bash theme={null}
curl "http://localhost:3040/api/tenants/019637b6-6d72-774b-9796-8c35813f9f78/todos"
```
You should see the following output
```bash theme={null}
{"message":"Unauthorized"}
```
Now, lets try to access the route with authentication. Make sure you replace the tenantId with the one you created in the previous step.
First, authenticate the user
```bash theme={null}
./get_cookies.sh spongebob@squarepants.com no-this-is-patrick
```
Then, access the route, using the cookies we got in the previous step
```bash theme={null}
curl "http://localhost:3040/api/tenants/019637b6-6d72-774b-9796-8c35813f9f78/todos" -b login_cookies.txt
```
You should see the following output
```bash theme={null}
[{"id":"019637b6-6d72-774b-9796-8c35813f9f78","title":"Finish Express integration","estimate":"2h","complete":false},{"id":"019637b6-6d72-774b-9796-8c35813f9f78","title":"Write documentation","estimate":"1h","complete":false}]
```
## Related Topics
## Next Steps
* [Learn more about Nile-Auth with Express](/auth/frameworks/express)
* [JWT Concepts](/auth/concepts/jwt)
* [Sessions](/auth/concepts/sessions)
* [Tenant Isolation](/tenant-virtualization/tenant-isolation)
* [SDK Reference](/auth/sdk-reference/javascript/overview)
# NextJs
Source: https://thenile.dev/docs/auth/quickstarts/nextjs
Get started with Nile Auth in minutes
This guide will help you get started with Nile Auth by walking you through the steps required to configure and integrate authentication in your application using the provided SDK components. By the end of this guide, you will have implemented email + signup authentication (enabled by default for all databases), user profile, and organization management in your application.
This guide uses Next.js with App Router, Typescript and Tailwind CSS. If you have a different framework in mind, you can find additional guides under "Frameworks"
in the sidebar. Initialize a new Next.js project with the following command and give it a name:
```bash theme={null}
npx create-next-app@latest nile-app --yes
```
1. If you haven't signed up for Nile yet, [sign up here](https://console.thenile.dev) and follow the steps to create a database.
2. Navigate to **Database Settings** in your database's UI at [console.thenile.dev](https://console.thenile.dev).
3. Go to **Connection** settings.
4. Select the CLI icon, and click **Generate credentials**
5. **Copy** the required credentials and **store them in an `.env` file** so they can be used in the application to connect to the Nile auth service.
```bash .env theme={null}
NILEDB_USER=niledb_user
NILEDB_PASSWORD=niledb_password
NILEDB_API_URL=https://us-west-2.api.thenile.dev/v2/databases/
NILEDB_POSTGRES_URL=postgres://us-west-2.db.thenile.dev:5432/
```
```bash theme={null}
npm install @niledatabase/server @niledatabase/react
@niledatabase/client @niledatabase/nextjs
```
Your application must expose API routes to handle authentication operations.
Create a folder called `api` under the `app` folder and a folder called `[...nile]` under it:
```bash theme={null}
mkdir -p app/api/\[...nile\]
```
Create following files handle the calls to your server, as well as expose the `nile` instance to your application:
```typescript app/api/[...nile]/nile.ts theme={null}
import { Nile } from '@niledatabase/server';
import { nextJs } from '@niledatabase/nextjs';
export const nile = Nile({ extensions: [nextJs] });
```
```typescript app/api/[...nile]/route.ts theme={null}
import { nile } from './nile';
export const { POST, GET, DELETE, PUT } = nile.handlers;
```
Your application will interact with above authentication routes using SDK components. Replace the boilerplate `app/page.tsx` with the following:
`/app/page.jsx`
```jsx theme={null}
import {
SignOutButton,
SignUpForm,
SignedIn,
SignedOut,
TenantSelector,
UserInfo,
} from "@niledatabase/react";
import "@niledatabase/react/styles.css";
export default function SignUpPage() {
return (
);
}
```
```bash theme={null}
npm run dev
```
Navigate to localhost to see the page. You should see a signup form that looks like this:
Enter a dummy email and password into the SignUpForm.
If all went well, you will be logged in automatically and see the user profile and an organization switcher that allows you to create new organizations and switch between them:
You can explore the database and the tenant management dashboard in the [Nile console](https://console.thenile.dev).
You should see a new organization and new user created in the "Tenants and Users" section.
You can also navigate the the query editor and view the user and tenats in the database tables directly by running:
```sql theme={null}
SELECT * FROM users;
SELECT * FROM tenants;
```
If you are feeling adventurous, you can try to replace the default user photo in their profile by running:
```sql theme={null}
UPDATE users
SET picture= 'https://www.publicdomainpictures.net/pictures/360000/velka/katze-katzchen-niedlich-vintage-1595638359oOf.jpg'
WHERE email='your-dummy-email@example.com';
```
You should see the user photo updated in the user profile in your application.
## Next Steps
* [Learn more about Nile-Auth with NextJS](/auth/frameworks/nextjs)
* [Learn About Backend Integration](/auth/frameworks/express)
# Remix
Source: https://thenile.dev/docs/auth/quickstarts/remix
Integrate Nile Auth with Remix applications
# Remix API Integration with Nile Database
This guide explains how to integrate **Nile Database** with **Remix** and set up routes for handling various HTTP requests (`GET`, `POST`, `PUT`, `DELETE`). Additionally, you'll see how to include **client-side components** for user authentication and interaction using **Nile's React SDK**.
***
Run the following command in your terminal to create a new Remix project:
```bash theme={null}
npx create-react-router@latest --template remix-run/react-router-templates/node-postgres
```
Follow the prompts and install your app. After creating the project, navigate into the newly created project directory:
```bash theme={null}
cd
```
```bash theme={null}
npm install @niledatabase/server @niledatabase/react @niledatabase/client
```
1. If you haven't signed up for Nile yet, [sign up here](https://console.thenile.dev) and follow the steps to create a database.
2. Navigate to **Database Settings** in your database's UI at [console.thenile.dev](https://console.thenile.dev).
3. Go to **Connection** settings.
4. Select the CLI icon, and click **Generate credentials**
5. **Copy** the required credentials and **store them in an `.env` file** so they can be used in the application to connect to the Nile auth service.
6. While you are there, click on the `PostgreSQL` icon and also **copy your database url** for drizzle to use
```bash theme={null}
NILEDB_USER=niledb_user
NILEDB_PASSWORD=niledb_password
NILEDB_API_URL=https://us-west-2.api.thenile.dev/v2/databases/
NILEDB_POSTGRES_URL=postgres://us-west-2.db.thenile.dev:5432/
DATABASE_URL=postgres://niledb_user:niledb_password0@us-west-2.db.thenile.dev:5432/
```
Now we must update the output from the default `create-react-router` for use with Nile. We want to switch to using `node-postgres`, and be able to do a top level `await` for nile configuration.
```bash bash [expandable] theme={null}
cat > server/app.ts << 'EOF'
import { createRequestHandler } from "@react-router/express";
import { drizzle } from "drizzle-orm/node-postgres";
import express from "express";
import postgres from "pg";
import "react-router";
import { DatabaseContext } from '~/database/context';
import * as schema from '~/database/schema';
declare module "react-router" {
interface AppLoadContext {
VALUE_FROM_EXPRESS: string;
}
}
export const app = express();
if (!process.env.DATABASE_URL) throw new Error("DATABASE_URL is required");
const client = new postgres.Client(process.env.DATABASE*URL);
await client.connect();
const db = drizzle(client, { schema });
app.use((*, \_\_, next) => DatabaseContext.run(db, next));
app.use(
createRequestHandler({
build: () => import("virtual:react-router/server-build"),
getLoadContext() {
return {
VALUE_FROM_EXPRESS: "Hello from Express",
};
},
})
);
EOF
cat > database/context.ts << 'EOF'
import { AsyncLocalStorage } from "node:async_hooks";
import type { NodePgDatabase } from "drizzle-orm/node-postgres";
import * as schema from './schema';
export const DatabaseContext = new AsyncLocalStorage<
NodePgDatabase
>();
export function database() {
const db = DatabaseContext.getStore();
if (!db) {
throw new Error("DatabaseContext not set");
}
return db;
}
EOF
cat > drizzle/0000_short_donald_blake.sql << 'EOF'
CREATE TABLE IF NOT EXISTS "guestBook" (
"id" integer PRIMARY KEY GENERATED ALWAYS AS IDENTITY,
"name" varchar(255) NOT NULL,
"email" varchar(255) NOT NULL,
CONSTRAINT "guestBook_email_unique" UNIQUE("email")
);
EOF
cat > vite.config.ts << 'EOF'
import { reactRouter } from "@react-router/dev/vite";
import tailwindcss from "@tailwindcss/vite";
import { defineConfig } from "vite";
import tsconfigPaths from "vite-tsconfig-paths";
export default defineConfig(({ isSsrBuild }) => ({
optimizeDeps: {
esbuildOptions: {
target: "esnext",
},
},
build: {
target: "esnext",
rollupOptions: isSsrBuild
? {
input: "./server/app.ts",
}
: undefined,
},
plugins: [tailwindcss(), reactRouter(), tsconfigPaths()],
}));
EOF;
```
Replace `/server/app.ts` with the following
```ts theme={null}
import { createRequestHandler } from "@react-router/express";
import { drizzle } from "drizzle-orm/node-postgres";
import express from "express";
import postgres from "pg";
import "react-router";
import { DatabaseContext } from "~/database/context";
import * as schema from "~/database/schema";
declare module "react-router" {
interface AppLoadContext {
VALUE_FROM_EXPRESS: string;
}
}
export const app = express();
if (!process.env.DATABASE_URL) throw new Error("DATABASE_URL is required");
const client = new postgres.Client(process.env.DATABASE_URL);
await client.connect();
const db = drizzle(client, { schema });
app.use((_, __, next) => DatabaseContext.run(db, next));
app.use(
createRequestHandler({
build: () => import("virtual:react-router/server-build"),
getLoadContext() {
return {
VALUE_FROM_EXPRESS: "Hello from Express",
};
},
})
);
```
Replace `database/context.ts` with the following
```ts theme={null}
import { AsyncLocalStorage } from 'node:async_hooks';
import type { NodePgDatabase } from 'drizzle-orm/node-postgres';
import * as schema from './schema';
export const DatabaseContext = new AsyncLocalStorage<
NodePgDatabase
>();
export function database() {
const db = DatabaseContext.getStore();
if (!db) {
throw new Error('DatabaseContext not set');
}
return db;
}
```
Update the placeholder table `drizzle/0000_short_donald_blake.sql` to generate correctly
```sql theme={null}
CREATE TABLE IF NOT EXISTS "guestBook" (
"id" integer PRIMARY KEY GENERATED ALWAYS AS IDENTITY,
"name" varchar(255) NOT NULL,
"email" varchar(255) NOT NULL,
CONSTRAINT "guestBook_email_unique" UNIQUE("email")
);
```
Modify `vite.config.ts` to allow for top level awaits
```ts theme={null}
import { reactRouter } from '@react-router/dev/vite';
import tailwindcss from '@tailwindcss/vite';
import { defineConfig } from 'vite';
import tsconfigPaths from 'vite-tsconfig-paths';
export default defineConfig(({ isSsrBuild }) => ({
optimizeDeps: {
esbuildOptions: {
target: 'esnext',
},
},
build: {
target: 'esnext',
rollupOptions: isSsrBuild
? {
input: './server/app.ts',
}
: undefined,
},
plugins: [tailwindcss(), reactRouter(), tsconfigPaths()],
}));
```
Now we need to add the nile instance and route handlers to allow our server to respond to authentication, user, and tenant requests.
```bash Bash [expandable] theme={null}
cat > app/nile.ts << 'EOF'
import { Nile } from "@niledatabase/server";
export const nile = Nile();
export const { handlers } = nile;
EOF;
cat > app/routes/nile-api.ts << 'EOF'
import type { Route } from "./+types/home";
import { handlers } from "~/nile";
const { GET, POST, PUT, DELETE } = handlers;
export const loader = async ({ request }: Route.LoaderArgs) => {
switch (request.method.toUpperCase()) {
case "GET":
return GET(request);
case "POST":
return POST(request);
case "PUT":
return PUT(request);
case "DELETE":
return DELETE(request);
default:
return new Response("Method Not Allowed", { status: 405 });
}
};
export const action = async ({ request }: Route.ActionArgs) => {
switch (request.method.toUpperCase()) {
case "POST":
return POST(request);
case "PUT":
return PUT(request);
case "DELETE":
return DELETE(request);
default:
return new Response("Method Not Allowed", { status: 405 });
}
};
EOF;
cat > app/routes.ts << 'EOF'
import { type RouteConfig, index, route } from "@react-router/dev/routes";
export default [
index("routes/home.tsx"),
route("api/*", "routes/nile-api.ts"),
] satisfies RouteConfig;
EOF
```
Create a file to house the main Nile instance. You can use this file to access nile from a central location any where in the app.
`app/nile.ts`
```ts theme={null}
import { Nile } from "@niledatabase/server";
export const nile = Nile();
export const { handlers } = nile;
```
Create the API route file at `app/routes/nile-api.ts`. This file will handle different HTTP methods (GET, POST, PUT, DELETE) using the **Nile SDK**.
```ts theme={null}
import type { Route } from './+types/home';
import { handlers } from '~/nile';
const { GET, POST, PUT, DELETE } = handlers;
export const loader = async ({ request }: Route.LoaderArgs) => {
switch (request.method.toUpperCase()) {
case 'GET':
return GET(request);
case 'POST':
return POST(request);
case 'PUT':
return PUT(request);
case 'DELETE':
return DELETE(request);
default:
return new Response('Method Not Allowed', { status: 405 });
}
};
export const action = async ({ request }: Route.ActionArgs) => {
switch (request.method.toUpperCase()) {
case 'POST':
return POST(request);
case 'PUT':
return PUT(request);
case 'DELETE':
return DELETE(request);
default:
return new Response('Method Not Allowed', { status: 405 });
}
};
```
This code handles different HTTP methods (`GET`, `POST`, `PUT`, `DELETE`) for the `/api/*` route and delegates the logic to Nile Database.
Update your routes to respond to the api
`app/routes.ts`
```ts theme={null}
import { type RouteConfig, index, route } from '@react-router/dev/routes';
export default [
index('routes/home.tsx'),
route('api/*', 'routes/auth-api.ts'),
] satisfies RouteConfig;
```
You can use the components from `@niledatabase/react` to handle authentication. Replace the boilerplate of the main `_index.tsx` file with the following:
This component will render:
* **User info** and the **guest book** if the user is signed in.
* **Sign-up form** if the user is not signed in.
```bash Bash [expandable] theme={null}
cat > app/routes/home.tsx << 'EOF'
import {
SignedIn,
SignedOut,
SignOutButton,
SignUpForm,
UserInfo,
} from "@niledatabase/react";
import '@niledatabase/react/styles.css';
import { database } from '~/database/context';
import * as schema from '~/database/schema';
import type { Route } from "./+types/home";
import { Welcome } from "../welcome/welcome";
export function meta({}: Route.MetaArgs) {
return [
{ title: "New React Router App" },
{ name: "description", content: "Welcome to React Router!" },
];
}
export async function action({ request }: Route.ActionArgs) {
const formData = await request.formData();
let name = formData.get("name");
let email = formData.get("email");
if (typeof name !== "string" || typeof email !== "string") {
return { guestBookError: "Name and email are required" };
}
name = name.trim();
email = email.trim();
if (!name || !email) {
return { guestBookError: "Name and email are required" };
}
const db = database();
try {
await db.insert(schema.guestBook).values({ name, email });
} catch (error) {
return { guestBookError: "Error adding to guest book" };
}
}
export async function loader({ context }: Route.LoaderArgs) {
const db = database();
const guestBook = await db.query.guestBook.findMany({
columns: {
id: true,
name: true,
},
});
return {
guestBook,
message: context.VALUE_FROM_EXPRESS,
};
}
export default function Home({ actionData, loaderData }: Route.ComponentProps) {
return (
);
}
EOF
```
Update the render of `home.tsx` to use the following components:
`app/routes/home.tsx`
```tsx theme={null}
import { SignedIn, SignedOut, SignUpForm, UserInfo } from "@niledatabase/react";
import "@niledatabase/react/styles.css";
{/**rest of the actions/meta/loader*/}
export default function Home({ actionData, loaderData }: Route.ComponentProps) {
return (
);
}
```
To run your project, execute the following:
```bash theme={null}
npm run db:migrate
npm run build
npm run dev
```
This will start the development server at `http://localhost:3000`, and you can test your API endpoints and authentication components.
## Summary
Now you can interact with your Nile Database through Remix API routes and manage authentication in your app!
## Related Topics
* [React Integration](/auth/frameworks/react)
* [Components](/auth/components/signin)
# Roadmap
Source: https://thenile.dev/docs/auth/roadmap
Future plans and upcoming features for Nile Auth
# JavaScript (client)
Source: https://thenile.dev/docs/auth/sdk-reference/javascript/client/authorizer
Use the Authorizer class from @niledatabase/client to manage authentication
## Installation
`@niledatabase/client` provides client-side functions for managing authentication, sessions, and user accounts. It is designed to work seamlessly with `@niledatabase/server` which provides the server-side routes.
```bash theme={null}
npm install @niledatabase/client
```
## Usage
The library exports a singleton instance of the `Authorizer` class, named `auth`, which is pre-configured for most use cases.
```ts theme={null}
import { auth } from '@niledatabase/client';
```
## Authentication
### signIn
Signs in a user. Supports both credentials (email/password) and Single Sign-On (SSO) providers.
```ts theme={null}
// Email and password
await auth.signIn('credentials', {
email: 'user@example.com',
password: 'password',
});
// Single Sign-On (Google, GitHub, etc.)
await auth.signIn('google', { callbackUrl: '/dashboard' });
```
#### Parameters
| Name | Type | Description |
| ---------- | -------- | ---------------------------------------------------------------------------- |
| `provider` | `string` | The authentication provider (e.g., `'credentials'`, `'google'`, `'github'`). |
| `options` | `object` | Configuration options. |
**Options:**
| Name | Type | Description |
| ------------- | --------- | ------------------------------------------------------------------------------------ |
| `email` | `string` | User's email (required for `'credentials'`). |
| `password` | `string` | User's password (required for `'credentials'`). |
| `callbackUrl` | `string` | URL to redirect to after successful sign-in. |
| `redirect` | `boolean` | If `false`, prevents automatic redirection (useful for handling responses manually). |
### signUp
Creates a new user account. Can optionally create a new tenant for the user.
```ts theme={null}
await auth.signUp({
email: 'user@example.com',
password: 'password',
createTenant: true, // Creates a tenant with the user's email as the name
});
```
#### Parameters
| Name | Type | Description |
| --------- | -------- | ---------------------- |
| `options` | `object` | Sign-up configuration. |
**Options:**
| Name | Type | Description |
| -------------- | ------------------- | ---------------------------------------------------------------------------------------------------------------- |
| `email` | `string` | User's email address. |
| `password` | `string` | User's password. |
| `createTenant` | `boolean \| string` | If `true`, creates a tenant named after the user's email. If a string, creates a tenant with that specific name. |
| `callbackUrl` | `string` | URL to redirect to after successful sign-up. |
### signOut
Logs the current user out and clears the session.
```ts theme={null}
await auth.signOut({ callbackUrl: '/goodbye' });
```
#### Parameters
| Name | Type | Description |
| --------- | -------- | ----------------------- |
| `options` | `object` | Sign-out configuration. |
**Options:**
| Name | Type | Description |
| ------------- | -------- | ---------------------------------- |
| `callbackUrl` | `string` | URL to redirect to after sign-out. |
## Session Management
### getSession
Retrieves the current user session. If a valid session is cached, it is returned; otherwise, a request is made to the server.
```ts theme={null}
const session = await auth.getSession();
if (session.user) {
console.log('User is logged in:', session.user);
}
```
## Password Management
### resetPassword
Initiates a password reset flow (typically by sending a reset email).
```ts theme={null}
await auth.resetPassword({
email: 'user@example.com',
password: 'new-pass', // Optional: if setting a new password directly
});
```
## Multi-factor Authentication
### mfa
Manages Multi-Factor Authentication (MFA) flows, including setup, challenge verification, and removal. The `User` object in the session includes a `multiFactor` property indicating if MFA is enabled.
```ts theme={null}
import { mfa } from '@niledatabase/client';
// or
// await auth.mfa({ ... });
```
#### Parameters
`mfa(params: MfaParams)` automatically selects the HTTP verb based on the payload.
| Name | Type | Default | Description |
| -------- | ---------------------------- | ----------------- | ------------------------------------------------------------------- |
| `method` | `'authenticator' \| 'email'` | `'authenticator'` | MFA method to target. |
| `scope` | `'setup' \| 'challenge'` | `'challenge'` | The operation scope. |
| `token` | `string` | `undefined` | Challenge token (required for verification and some disable flows). |
| `code` | `string` | `undefined` | 6-digit verification code or recovery code. |
| `remove` | `boolean` | `false` | If `true`, disables MFA for the specified method. |
#### Examples
```ts setup theme={null}
// 1. Start Setup
// Returns a payload with secret/otpauthUrl (for QR) or triggers an email code
const setup = await auth.mfa({ method: 'authenticator', scope: 'setup' });
```
```ts verify theme={null}
// 2. Verify Challenge
// Use the token from setup or login challenge, and the user's code
const verified = await auth.mfa({
token: setup.token,
code: '123456',
method: setup.method,
scope: 'challenge',
});
```
```ts disable theme={null}
// 3. Disable MFA
await auth.mfa({ method: 'authenticator', remove: true });
```
```ts challenge theme={null}
// 4. Handle Login Challenge
const challenge = await auth.signIn('credentials', { email, password });
// If MFA is required, the response contains a token
if (challenge && typeof challenge === 'object' && 'token' in challenge) {
const result = await auth.mfa({
token: String(challenge.token),
code: '654321', // Code input by user
method: challenge.method ?? 'authenticator',
scope: 'challenge',
});
if (!result?.ok) throw new Error('MFA verification failed');
}
```
#### Error Handling
Non-200 responses may return a `ChallengeRedirect` object (`{ url: string }`) with an `error` query parameter. Parse this parameter to display user-friendly error messages.
## Utilities
### getCsrfToken
Retrieves a CSRF token for form submissions.
```ts theme={null}
const token = await auth.getCsrfToken();
// Optional: Fetch from custom URL
const customToken = await auth.getCsrfToken('/api/auth/custom-csrf');
```
### getProviders
Retrieves the list of configured authentication providers.
```ts theme={null}
const providers = await auth.getProviders();
```
## Advanced Configuration
### Custom Authorizer
For advanced use cases, such as connecting to a different API endpoint or managing multiple auth contexts, you can create a custom instance of `Authorizer`.
```ts theme={null}
import { Authorizer } from '@niledatabase/client';
const customAuth = new Authorizer({
baseUrl: 'https://api.myapp.com',
basePath: '/auth',
});
```
#### Constructor Options
| Name | Type | Description |
| ---------- | ------------- | ----------------------------------------- |
| `baseUrl` | `string` | The base URL of the API. |
| `basePath` | `string` | Path to auth endpoints (default: `/api`). |
| `init` | `RequestInit` | Default fetch options for all requests. |
The instance exposes a `state` object with `baseUrl`, `session`, and loading status. Configuration can be updated later via `auth.configure({ ... })`.
# Overview
Source: https://thenile.dev/docs/auth/sdk-reference/javascript/overview
Toolkit for building multi-tenant applications with Next.js, Remix, Express, and more
## Available Modules
The SDK provides a number of modules with methods for interacting with Nile:
Securely authenticating users
Tenant-aware querying of your database
Managing users
Managing tenants
Built-in and custom routes
## Understanding the Nile Instance
A configured Nile instance handles proxying requests to the Nile Auth API. Your client connects to your server, which then formats and ensures
the requests can be handled by the Nile Auth API. The SDK is designed to make clients talking through your server to the Nile auth service as
transparent as possible. In addition, you can use the SDK to directly access the Nile Auth API from your server.
## A Note on Context
It is important to ensure that the proper context is being used when calling functions from the server side. This can be done in [extensions](../../concepts/extensions) intelligently, or by using `nile.withContext`.
Additionally, the context for the database is enforced for calls via `nile.query(...)`, but not for `nile.db.query(...)`.
The SDK will try its hardest to use context whenever possible. Under the hood, it uses `AsyncLocalStorage`, which relies on execution within the same request lifecycle. If you await or dispatch across async boundaries without using the SDK, the context may be lost, in which will default to the last used.
The easiest way to deal with this and ensure proper context 100% of the time is to use a cookie saved by your application when a user determines their tenant, and the SDK will automatically look for `nile.tenant-id` cookie (`TENANT_COOKIE` exported by `@niledatabase/server`).
## Authentication Flow
### 1. User Initiates Authentication
* The user clicks a "Sign in" button.
* This action triggers a `signIn` method with the chosen provider. Your server handles all requests, which in most cases is simply forwarding them on to the Nile auth service with some additional information to help identify the client.
### 2. Redirect to Provider (OAuth Flow)
* If an OAuth provider (e.g., Google, GitHub) is used, the user is redirected to the provider's authentication page. This works by Nile auth returning redirects to your application, which the SDK handles in order to send the user to the provider.
* The user enters their credentials and grants permission to the application. Because your server is handling the requests, the user is redirected back to your application.
### 3. Provider Callback & Token Exchange
* After successful authentication, the provider redirects the user back to your application, which proxies the request to the Nile auth service.
* Nile auth exchanges the authorization code for an access token and forwards the authorization information to your server, which in turn would just pass that on to the client.
### 4. Session Creation
* Via your service, nile auth provides a secure cookie.
* The cookie includes basic user information, which can be accessed using the `nile.auth.getSession` or a full user profile via `nile.users.getSelf`
### 5. Accessing the Session
* A session is always within the context of a request. For some frameworks, the session management is handled by an extension, but you can always session data by forwarding the client's request:
```typescript JS theme={null}
import http from "http";
import { Nile } from "@niledatabase/server";
const nile = Nile();
const server = http.createServer(async (req, res) => {
const session = await nile.auth.getSession(req);
if (session) {
console.log("User is authenticated", session.user);
});
)}
```
```typescript nextjs theme={null}
import { headers } from "next/headers";
import { Nile } from "@niledatabase/server";
import { nextJs } from "@niledatabase/nextjs";
const nile = Nile({ extensions: [nextJs] });
export default function Page() {
// without the extension, you would use nile.withContext({ headers: await headers() })
const session = await nile.auth.getSession();
}
```
## API Reference
The API reference is available in the [API Reference](/auth/api-reference/auth/sign-in-to-the-application) section.
# Auth
Source: https://thenile.dev/docs/auth/sdk-reference/javascript/server/auth
Secure server-side user authentication
The Auth module provides a comprehensive interface for managing user authentication and sessions on the server side. It wraps all interactions with the `/api/auth` endpoints including sign-up, sign-in, session handling, password reset, and provider management.
This module is intended for use in server-side code only. For client-side login flows, use the React SDK components or OAuth redirects.
All Auth methods depend on cookies and headers for managing CSRF tokens,
sessions, and callback behavior. For framework-specific integrations (e.g.,
Next.js), some of these flows may be automated.
***
## signUp
Create a new user and start a session using email/password credentials.
```ts signUp theme={null}
const user = await nile.auth.signUp({
email: 'jane@example.com',
password: 'secure123',
});
```
```ts signUp raw theme={null}
const res = await nile.auth.signUp(
{
email: 'jane@example.com',
password: 'secure123',
},
true,
);
```
### Parameters
* `email`: required string
* `password`: required string
* `tenantId`: optional. Join an existing tenant
* `newTenantName`: optional. If no `tenantId` **is** provided, creates a new tenant with this name
### Returns
* A `User` object (with associated tenant info) or a raw `Response`
On success, a session token is automatically stored via `withContext()` and
used in subsequent calls.
***
## signIn
Authenticate an existing user with email/password or via a configured provider.
```ts credentials theme={null}
const user = await nile.auth.signIn('email', {
email: 'jane@example.com',
password: 'secure123',
});
```
```ts raw theme={null}
const res = await nile.auth.signIn(
'email',
{
email: 'jane@example.com',
password: 'secure123',
},
true,
);
```
### Parameters
* `provider`: either `'email'` (for credentials) or another supported provider name
* `payload`: either a request or credentials object
* `rawResponse`: optional, when `true` returns a `Response`
### Returns
* The authenticated `User` object or a `Response`
This method handles both OAuth and email/password sign-in flows. For OAuth
providers, the client should call `/api/auth/signin/{provider}`.
***
## signOut
Ends the current session.
```ts theme={null}
await nile.auth.signOut();
```
### Returns
* A `Response` object from `/api/auth/signout`
On success, the session and CSRF cookies are cleared, and `nile.withContext()` resets the current session state.
This method is only for server-side sign out. To clear browser cookies, use
the React SDK ` ` component.
***
## getSession
Retrieve the current user session.
```ts theme={null}
const session = await nile.auth.getSession();
```
### Returns
Either a `JWT` or an `ActiveSession` object, depending on the session type:
```ts JWT theme={null}
{
email: string;
sub: string;
id: string;
iat: number;
exp: number;
jti: string;
}
```
```ts ActiveSession theme={null}
{
id: string;
email: string;
expires: string;
user?: {
id: string;
name: string;
image: string;
email: string;
emailVerified: void | Date;
};
}
```
Returns `undefined` if no session is active.
***
## getCsrf
Fetches the CSRF token used for protected requests.
```ts theme={null}
const csrf = await nile.auth.getCsrf();
```
### Returns
A `csrfToken` string or a `Response` if requested
This method is called automatically by `signIn`, `signUp`, and
`resetPassword`. You typically don’t need to call it manually.
***
## listProviders
Retrieve the list of authentication providers configured for the current tenant.
```ts theme={null}
const providers = await nile.auth.listProviders();
```
### Returns
A mapping of provider names to provider configs:
```ts theme={null}
{
[provider: string]: {
id: string;
name: string;
type: string;
signinUrl: string;
callbackUrl: string;
}
}
```
***
## forgotPassword
Initiate a password reset request.
```ts theme={null}
await nile.auth.forgotPassword({
email: 'jane@example.com',
callbackUrl: 'https://myapp.com/reset-confirm',
});
```
### Parameters
* `email`: required string
* `callbackUrl`: optional redirect after password reset
* `redirectUrl`: optional link shown in the email
### Returns
A `Response` with password reset instructions
***
## resetPassword
Complete the password reset flow.
```ts theme={null}
await nile.auth.resetPassword({
email: 'jane@example.com',
password: 'newPassword123',
});
```
### Parameters
* `email`, `password`: required
* `callbackUrl`, `redirectUrl`: optional fallback URLs
### Returns
A `Response`. On success, session headers are automatically applied using `withContext()`.
***
## callback
Handle a callback from an OAuth provider.
```ts theme={null}
await nile.auth.callback('github', request);
```
### Parameters
* `provider`: name of the auth provider
* `body`: a `Request` or serialized request body
### Returns
A `Response` with session cookies or error headers
This method is rarely used directly. It powers the internal OAuth flow of
`signIn()`.
***
## Multi-factor (Server)
Server-side helpers for enrolling, challenging, and removing MFA in Nile Auth.
### Features
* Single `auth.mfa` entry point for setup, verification, and removal against `/auth/mfa`
* Works with either authenticator apps or email one-time codes
* Reuses the server context (`withContext`) so CSRF, session, and callback cookies flow automatically
* Returns parsed JSON by default while allowing raw `Response` access when needed
* Redirect-friendly responses for server-rendered flows (e.g., Next.js middleware)
### Installation
```sh theme={null}
npm install @niledatabase/server
```
### mfa
Server entry point for `/api/auth/mfa`. HTTP verbs map automatically based on params.
```ts setup theme={null}
// Start authenticator enrollment (POST /auth/mfa)
const setup = await nile.auth.mfa({
scope: 'setup',
method: 'authenticator',
});
```
```ts verify theme={null}
// Verify a setup or login challenge (PUT /auth/mfa)
const verified = await nile.auth.mfa({
token: setup.token,
code: '123456',
scope: setup.scope,
method: setup.method,
});
```
```ts remove theme={null}
// Remove MFA (DELETE /auth/mfa)
await nile.auth.mfa({
method: 'authenticator',
remove: true,
});
```
```ts sign-in challenge theme={null}
// Handle MFA required after sign-in
const challenge = await nile.auth.signIn('email', {
email: 'user@example.com',
password: 'correct-horse',
});
if (challenge && typeof challenge === 'object' && 'token' in challenge) {
await nile.auth.mfa({
token: String((challenge as any).token),
code: '654321',
scope: 'challenge',
method: (challenge as any).method ?? 'authenticator',
});
}
```
#### Parameters
* `token`: string. MFA challenge token issued during setup or sign-in; required for `scope: 'challenge'`.
* `scope`: `'setup' | 'challenge'`. Indicates setup vs. completing a challenge. Defaults to `'challenge'`.
* `method`: `'authenticator' | 'email'`. MFA mechanism to operate on. Defaults to `'authenticator'`.
* `code`: string. Verification or recovery code when completing a challenge.
* `remove`: boolean. When `true`, sends `DELETE` to remove the current method. Defaults to `false`.
* `rawResponse`: boolean. When `true`, returns the raw `Response` instead of parsed JSON.
#### Behavior
* `POST` to initiate setup, `PUT` with a `token` to verify, and `DELETE` when `remove` is `true`.
* Runs inside `withNileContext`, so cookies/CSRF from prior calls are reused automatically.
* Redirects surface as `{ url: string }` with an `error` query param; otherwise expect API status codes (400 invalid payload, 401 invalid code/unauthenticated, 403 token mismatch, 404 missing challenge, 410 expired challenge, 500 server error).
#### Response shapes
* Authenticator setup: `{ method: 'authenticator'; token: string; scope: 'setup' | 'challenge'; otpauthUrl?: string; secret?: string; recoveryKeys?: string[] }`
* Email setup: `{ method: 'email'; token: string; scope: 'setup' | 'challenge'; maskedEmail?: string }`
* Challenge/verification: `{ ok: true; scope: 'setup' | 'challenge'; recoveryCodesRemaining?: number }`
* Redirects/errors: `{ url: string }`
#### Tips
* Persist and reuse the `token` returned from setup or sign-in for all follow-up calls.
* For authenticated server-rendered pages, call `await nile.auth.getSession()` after `auth.mfa` so session cookies are hydrated in the current context.
* Admin removal flows may return a challenge; route the user to MFA verification with the provided `token` instead of bypassing MFA.
* Codes are typically 6 digits; recovery codes are string tokens issued during setup.
***
## Utility Methods
These are internal helpers that power the core authentication flows:
### parseToken
Extracts the session token from headers.
```ts theme={null}
const token = parseToken(response.headers);
```
### parseCallback
Extracts the callback URL cookie from headers.
```ts theme={null}
const cbUrl = parseCallback(request.headers);
```
### parseResetToken
Extracts the reset token from password reset response.
```ts theme={null}
const resetToken = parseResetToken(response.headers);
```
These functions are mostly used internally. You typically don't need to
call them directly unless customizing the authentication flow.
\| Name | Type | Default | Description | | --- | --- | --- | --- | | `token` |
`string` | `undefined` | MFA challenge token issued during setup or sign-in;
required for `scope: 'challenge'`. | | `scope` | `'setup' \| 'challenge'` |
`'challenge'` | Indicates whether you are starting setup or completing a
challenge. | | `method` | `'authenticator' \| 'email'` | `'authenticator'` | MFA
mechanism to operate on. | | `code` | `string` | `undefined` | Verification or
recovery code when completing a challenge. | | `remove` | `boolean` | `false` |
Sends a `DELETE` request to remove the currently enrolled method. | |
`rawResponse` | `boolean` | `false` | When `true`, skip JSON parsing and return
the raw `Response`. |
Behavior:
* HTTP verb selection follows the API spec: `POST` to initiate setup, `PUT` with a `token` to verify challenges, and `DELETE` when `remove` is `true`. Ensure you pass the correct `scope`/`token`/`remove` combination so the helper sends the intended verb; if you call the HTTP API directly, use the same verbs described above.
* The helper runs inside `withNileContext`, so cookies and CSRF headers from prior calls (e.g., `signIn`, `getCsrf`) are reused automatically.
* Errors are returned as `{ url: string }` with an `error` query parameter when the upstream responds with a redirect; otherwise expect HTTP status codes surfaced from the API (400 invalid payload, 401 invalid code or unauthenticated, 403 token mismatch, 404 not found, 410 expired challenge, 500 server error).
### Response shapes
* Authenticator setup: `{ method: 'authenticator'; token: string; scope: 'setup' \| 'challenge'; otpauthUrl?: string; secret?: string; recoveryKeys?: string[] }`
* Email setup: `{ method: 'email'; token: string; scope: 'setup' \| 'challenge'; maskedEmail?: string }`
* Challenge/verification: `{ ok: true; scope: 'setup' \| 'challenge'; recoveryCodesRemaining?: number }`
* Redirects/errors: `{ url: string }` (inspect the `error` search param for the reason)
### Tips
* Always persist and reuse the `token` returned from setup or sign-in; it must be included on all follow-up MFA calls.
* For authenticated server-rendered pages, call `await nile.auth.getSession()` after `auth.mfa` to ensure session cookies are hydrated in the current context.
* When building admin tooling to disable MFA, expect a challenge response; route the user to your verification UI with the returned `token` instead of bypassing MFA.
* Authenticator/email codes are expected to be 6 digits; recovery codes are string tokens issued during setup. An expired or missing challenge returns 410/404, and an invalid code returns 401.
# Configuration
Source: https://thenile.dev/docs/auth/sdk-reference/javascript/server/configuration
Configuration options for the Nile-JS SDK
NileJS SDK can be configured using both environment variables and a configuration object.
If both are provided, the configuration object will take precedence.
This document describes the available configuration options, and how to set them using either method.
## SDK Configuration
The following configuration options are available at the overall SDK level:
| Configuration Option | Environment Variable | Required | Default | Description |
| -------------------- | -------------------------------------- | -------- | ------------------ | ---------------------------------------------------------------------------------- |
| `databaseId` | `NILEDB_ID` or `NILEDB_API_URL` | Yes | - | The ID of the database to use |
| `user` | `NILEDB_USER` | Yes | - | The user to use for the database |
| `password` | `NILEDB_PASSWORD` | Yes | - | The password to use for the database |
| `databaseName` | `NILEDB_NAME` or `NILEDB_POSTGRES_URL` | Yes | - | The name of the database to use |
| `tenantId` | `NILEDB_TENANT` | No | - | Current tenant ID used for scoping API and DB calls |
| `userId` | - | No | - | Optional user identifier used for DB access, e.g. for impersonation |
| `debug` | - | No | `false` | Whether to enable debug logging |
| `apiUrl` | `NILEDB_API_URL` | No | - | Base URL for Nile Auth requests |
| `callbackUrl` | `NILEDB_CALLBACK_URL` | No | - | Used to override the client-provided callback URL during authentication |
| `origin` | - | No | - | Controls origin used in requests (e.g. for cookie handling in cross-origin apps) |
| `headers` | - | No | - | Additional headers sent with API requests. Include `cookie` for session management |
| `routes` | - | No | - | Overrides for default auth routes (e.g. signin path) |
| `routePrefix` | - | No | `/api` | Prefix for API routes |
| `secureCookies` | `NILEDB_SECURECOOKIES` | No | Based on NODE\_ENV | Enforces use of secure cookies |
| `extensions` | - | No | - | Array of pre/post-processing hooks for each API request |
| `preserveHeaders` | - | No | false | Whether to retain incoming headers across extension execution |
| `logger` | - | No | - | Optional custom logger implementation |
| `db` | - | No | - | Database connection pool configuration (see below) |
## Database Configuration
The `db` object configures the connection to Postgres, using settings compatible with the `pg` library.
| Configuration Option | Environment Variable | Required | Default | Description |
| --------------------------------------- | -------------------------------------- | -------- | ---------------- | -------------------------------------------------------- |
| user | `NILEDB_USER` | No | - | Will default to the top-level `user` config if not set |
| password | `NILEDB_PASSWORD` | No | - | Will default to the top-level `password` if not set |
| host | `NILEDB_HOST` or `NILEDB_POSTGRES_URL` | No | `db.thenile.dev` | Postgres host |
| port | `NILEDB_PORT` or `NILEDB_POSTGRES_URL` | No | 5432 | Postgres port |
| database | `NILEDB_NAME` or `NILEDB_POSTGRES_URL` | No | - | Postgres database name |
| connectionString | - | No | - | Full connection string. Overrides other DB config values |
| ssl | - | No | - | TLS socket options for secure DB connection |
| types | - | No | - | Custom Postgres type parsers |
| statement\_timeout | - | No | - | Statement timeout in ms |
| query\_timeout | - | No | - | Query timeout in ms |
| lock\_timeout | - | No | - | Lock timeout in ms |
| application\_name | - | No | - | Name used by the DB client connection |
| connectionTimeoutMillis | - | No | - | How long to wait before failing a DB connection attempt |
| idle\_in\_transaction\_session\_timeout | - | No | - | Idle timeout for sessions with open transactions |
| idleTimeoutMillis | - | No | 10000 | Time a client can sit idle before being dropped |
| max | - | No | 10 | Max number of DB clients in the pool |
| allowExitOnIdle | - | No | false | Allow Node to exit early when pool is idle |
## Example Configurations
### Basic configuration with environment variables
```bash .env theme={null}
NILEDB_USER=0195f45d-c3ca-7b29-8a9c-717a99d993a1
NILEDB_PASSWORD=5e64c8e9-8ac7-4c0d-b6f1-dab7672fe1be
NILEDB_API_URL=https://us-west-2.api.thenile.dev/v2/databases/0195f4dd-22d3-68c6-a6c1-1ddc24f0f37c
NILEDB_POSTGRES_URL=postgres://us-west-2.db.thenile.dev:5432/my_database
```
```ts nile.ts theme={null}
import { Nile } from '@niledatabase/server';
const nile = Nile();
```
### Basic configuration with configuration object
```ts nile.ts theme={null}
import { Nile } from '@niledatabase/server';
const nile = Nile({
databaseId: '0195f4dd-22d3-68c6-a6c1-1ddc24f0f37c',
databaseName: 'my_database',
user: '0195f45d-c3ca-7b29-8a9c-717a99d993a1',
password: '5e64c8e9-8ac7-4c0d-b6f1-dab7672fe1be',
apiUrl:
'https://us-west-2.api.thenile.dev/v2/databases/0195f4dd-22d3-68c6-a6c1-1ddc24f0f37c',
db: {
host: 'us-west-2.db.thenile.dev',
},
});
```
### Production configuration
```bash .env theme={null}
NILEDB_USER=0195f45d-c3ca-7b29-8a9c-717a99d993a1
NILEDB_PASSWORD=5e64c8e9-8ac7-4c0d-b6f1-dab7672fe1be
NILEDB_API_URL=https://us-west-2.api.thenile.dev/v2/databases/0195f4dd-22d3-68c6-a6c1-1ddc24f0f37c
NILEDB_POSTGRES_URL=postgres://us-west-2.db.thenile.dev:5432/my_database
DEBUG=false
```
```ts nile.ts theme={null}
import { Nile } from '@niledatabase/server';
const nile = Nile({
debug: process.env.DEBUG === 'true',
db: {
max: 20,
idleTimeoutMillis: 30000,
},
});
```
### Cross-origin configuration
```bash .env theme={null}
NILEDB_USER=0195f45d-c3ca-7b29-8a9c-717a99d993a1
NILEDB_PASSWORD=5e64c8e9-8ac7-4c0d-b6f1-dab7672fe1be
NILEDB_API_URL=https://us-west-2.api.thenile.dev/v2/databases/0195f4dd-22d3-68c6-a6c1-1ddc24f0f37c
NILEDB_POSTGRES_URL=postgres://us-west-2.db.thenile.dev:5432/my_database
```
```ts nile.ts theme={null}
import { Nile } from '@niledatabase/server';
const nile = Nile({
origin: 'https://my-frontend.com',
});
```
# DB
Source: https://thenile.dev/docs/auth/sdk-reference/javascript/server/db
Using the Nile-JS SDK DB client
The `db` object is used to query the database. It is designed as a tenant-aware connection pool.
When `nile.tenantId` is set, the `db` object will use the correct tenant's virtual database.
## query
The `query` method is used to query the database. This is the main method exposed by the `db` object.
### Parameters
* `sql`: The SQL query to execute.
* `params`: The parameters to use in the query.
### Querying a shared table
All tenants and users have access to any [shared tables](/tenant-virtualization/tenant-sharing) - tables without a tenant id column.
```ts theme={null}
const db = nile.db;
await db.query(
'CREATE TABLE IF NOT EXISTS shared_table (id INTEGER, name TEXT)',
);
await db.query('INSERT INTO shared_table (id, name) VALUES (1, "John Doe")');
const result = await db.query('SELECT * FROM shared_table');
console.log(result);
```
### Querying a tenant-aware table
[Tenant-aware tables](/tenant-virtualization/tenant-isolation) have a tenant id column.
If you query without setting `nile.tenantId`, this will be a cross-tenant query.
If you set `nile.withContext({ tenantId })`, the query will run against the tenant's virtual database.
When inserting data into a tenant-aware table, you need to provide the tenant id - and it has to belong to an existing tenant.
If you set `nile.withContext({ tenantId })`, the tenant id must match the tenant id in the table.
In the example below, we also use parameter substitution to prevent SQL injection.
`nile.db` is a context-less way to query the database. `nile.query` uses
context when querying the database, if the context is set
```ts cross-tenant query theme={null}
const { db } = nile; // using DB means it will be a cross-tenant query.
const some_name = 'John Doe';
const some_tenant_id = '0195f45d-c3ca-7b29-8a9c-717a99d993a1';
await db.query(
'CREATE TABLE IF NOT EXISTS tenant_table (id INTEGER, name TEXT, tenant_id UUID)',
);
await db.query(
'INSERT INTO tenant_table (id, name, tenant_id) VALUES (1, $1, $2)',
[some_name, some_tenant_id],
);
const result = await db.query('SELECT * FROM tenant_table'); // this will return all rows from all tenants
console.log(result);
```
```ts tenant-aware query theme={null}
const context = {
tenantId: '0195f45d-c3ca-7b29-8a9c-717a99d993a1'; // may be omitted if already set
}
const { query } = await nile.withContext(context);
const some_name = 'John Doe';
await query('CREATE TABLE IF NOT EXISTS tenant_table (id INTEGER, name TEXT, tenant_id UUID)');
await query('INSERT INTO tenant_table (id, name, tenant_id) VALUES (1, $1, $2)', [some_name , context.tenantId]);
const result = await query('SELECT * FROM tenant_table'); // this will only return rows where tenant_id matches the current tenant
console.log(result);
```
## client
The `client` method is used to get a client from the database connection pool. This can be useful if you want to run several queries on the same connection.
The client is not automatically released back to the pool after the query is
executed. You need to call `client.release()` to release it back to the pool.
### Example
The example below uses `client` to run a transaction - a group of queries that are executed as a single unit.
If any of the queries fail, the transaction is rolled back.
If all queries succeed, the transaction is committed.
The `client` object guarantees that all queries are executed on the same connection.
It is then released back to the pool after the transaction is committed or rolled back.
```ts theme={null}
const { db } = nile;
const client = await db.client();
try {
await client.query('BEGIN');
await db.query('INSERT INTO shared_table (id, name) VALUES (1, "John Doe")');
await db.query('INSERT INTO shared_table (id, name) VALUES (2, "Jane Doe")');
await client.query('COMMIT');
} catch (error) {
await client.query('ROLLBACK');
throw error;
} finally {
await client.release();
}
```
# Routes
Source: https://thenile.dev/docs/auth/sdk-reference/javascript/server/routes
Routes in the Nile-JS SDK
The Nile-JS SDK includes generated routes for all its operations.
These routes are used by the SDK methods to proxy requests to nile-auth, as well as directly from the React hooks and components in `@niledatabase/react`.
## Generating routes
Route generation depends on the framework you are using.
### Next.js example
```ts nile.ts theme={null}
// app/api/[...nile]/nile.ts
import { Nile } from "@niledatabase/server";
export const nile = Nile();
export const { handlers } = nile;
```
```ts route.ts theme={null}
// app/api/[...nile]/route.ts
import { handlers } from './nile';
export const { POST, GET, DELETE, PUT } = handlers;
```
### Remix example
```ts nile.ts theme={null}
// app/nile.ts
import { Nile } from "@niledatabase/server";
export const nile = Nile();
export const { handlers } = nile;
```
```ts nile-api.ts theme={null}
//app/routes/nile-api.ts
// This is where the routes are exported
import type { Route } from './+types/home';
import { handlers } from '~/nile';
const { GET, POST, PUT, DELETE } = handlers;
export const loader = async ({ request }: Route.LoaderArgs) => {
switch (request.method.toUpperCase()) {
case 'GET':
return GET(request);
case 'POST':
return POST(request);
case 'PUT':
return PUT(request);
case 'DELETE':
return DELETE(request);
default:
return new Response('Method Not Allowed', { status: 405 });
}
};
export const action = async ({ request }: Route.ActionArgs) => {
switch (request.method.toUpperCase()) {
case 'POST':
return POST(request);
case 'PUT':
return PUT(request);
case 'DELETE':
return DELETE(request);
default:
return new Response('Method Not Allowed', { status: 405 });
}
};
```
```ts routes.ts theme={null}
// app/routes.ts
// This routes all the api/* requests to the routes in nile-api.ts
import { type RouteConfig, index, route } from '@react-router/dev/routes';
export default [
index('routes/home.tsx'),
route('api/*', 'routes/nile-api.ts'),
] satisfies RouteConfig;
```
### Express example
```ts server.mjs theme={null}
import "dotenv/config";
import express from "express";
import { Nile } from "@niledatabase/server";
import { NileExpressHandler } from "@niledatabase/server/express";
const startServer = async () => {
try {
const app = express();
const nile = Nile();
// This is where the route handlers are imported
const { paths, handler } = await NileExpressHandler(nile);
app.use(express.json());
app.use(express.urlencoded({ extended: true }));
// This is where the routes are exposed in your express app
app.get(paths.get, handler);
app.post(paths.post, handler);
app.put(paths.put, handler);
app.delete(paths.delete, handler);
// Your own routes go here
const PORT = process.env.PORT || 3040;
app.listen(PORT, () => {
console.log(`Server is running on port ${PORT}`);
});
} catch (error) {
console.error("Error starting server:", error);
process.exit(1);
}
};
startServer();
```
## Using routes
You typically don't need to use the routes directly.
The SDK methods and react/web components use the routes under the hood to communicate with nile-auth.
The generated routes are available via `nile.paths`. For reference, here are the paths for the default routes:
```ts theme={null}
export const appRoutes = (prefix = '/api'): Routes => ({
SIGNIN: `${prefix}/auth/signin`,
PROVIDERS: `${prefix}/auth/providers`,
SESSION: `${prefix}/auth/session`,
CSRF: `${prefix}/auth/csrf`,
CALLBACK: `${prefix}/auth/callback`, // this path has a route per enabled provider (e.g. google, github, etc.)
SIGNOUT: `${prefix}/auth/signout`,
ERROR: `${prefix}/auth/error`,
VERIFY_REQUEST: `${prefix}/auth/verify-request`,
PASSWORD_RESET: `${prefix}/auth/reset-password`,
ME: `${prefix}/me`,
USERS: `${prefix}/users`,
TENANTS: `${prefix}/tenants`,
TENANT: `${prefix}/tenants/{tenantId}`,
TENANT_USER: `${prefix}/tenants/{tenantId}/users/{userId}`,
TENANT_USERS: `${prefix}/tenants/{tenantId}/users`,
SIGNUP: `${prefix}/signup`,
LOG: `${prefix}/_log`,
});
```
A case where you might want to use the routes directly is when these routes are exposed as a REST API of your application backend.
For example, if you use the `@niledatabase/server/express` package, the routes are exposed in the `app` object and
while you may have a frontend that uses `@niledatabase/react` to call these routes from the application UI,
you may want to also use them as a REST API for another backend or external service.
In this case, you need to use the routes directly. The key is to:
1. Include both the session cookie and the CRSF cookie in the request headers.
2. Include the CSRF token in the request body.
Here are a few examples of how to call the routes directly:
```bash getCSRFToken theme={null}
# This gets the CSRF token from an API exposed by your application, and also saves the cookies to a file
csrf_token=$(curl -s -X GET "http://localhost:3040/api/auth/csrf" -c "csrf_cookies.txt" | jq -r '.csrfToken')
```
```bash signup theme={null}
# This signs up a new user with email/password credentials
# The CSRF token is included in the request body and in the cookies
# The cookies are saved to a file so they can be used in subsequent requests
curl -X POST "http://localhost:3040/api/signup" \
-H "Content-Type: application/json" \
-b "csrf_cookies.txt" \
--cookie-jar "login_cookies.txt" \
-d "{\"csrfToken\":\"$csrf_token\",\"email\":\"newuser@example.com\",\"password\":\"foobar\"}"
```
```bash login theme={null}
# This logs in a user with email/password credentials
# The CSRF token is included in the request body and in the cookies
# The cookies are saved to a file so they can be used in subsequent requests
curl -X POST "http://localhost:3040/api/auth/callback/credentials" \
-H "Content-Type: application/json" \
-b "csrf_cookies.txt" \
--cookie-jar "login_cookies.txt" \
-d "{\"csrfToken\":\"$csrf_token\",\"email\":\"newuser@example.com\",\"password\":\"foobar\"}"
```
```bash getMe theme={null}
# This gets the user's profile information
curl -X GET "http://localhost:3040/api/me" -b "login_cookies.txt"
```
```bash createTenant theme={null}
# This creates a new tenant
curl -X POST 'localhost:3040/api/tenants' \
-H 'Content-Type: application/json' \
-d '{"name":"my first customer"}' \
-b login_cookies.txt
```
## Overriding routes
Sometimes you might want to intercept or override the routes used by the SDK in order to inject your own logic.
For example, adding your own logging or metrics, adding debugging information, or perhaps injecting your own cookies during login.
There are three ways to override routes:
1. **Route wrappers**: Wrap the route in your own logic. This can be done in the routes file, and is useful for minor modifications or debugging.
2. **Route overrides**: Override the route for a specific operation.
### Route wrappers
In the examples below, we'll use route wrappers to log the headers before every request and the body if it's a POST request.
We'll also log the status code of the response.
```ts NextJS route wrappers theme={null}
// app/api/[...nile]/route.ts
import { handlers } from "./nile";
// Middleware function to log request and response details
const logRequestAndResponseDetails = (handler) => async (req, res) => {
// Log the request method and URL
console.log(`Request Method: ${req.method}, Request URL: ${req.url}`);
// Log the request headers
console.log('Request Headers:', req.headers);
// Clone the request to safely read the body
const clonedReq = req.clone();
// Log the request body if it's a POST or PUT request
if (req.method === 'POST') {
const body = await clonedReq.text();
console.log('Request Body:', body);
}
// Call the original handler and return its result
const result = await handler(req, res);
// Log the response status after the handler has executed
console.log('Result Status:', result.status);
return result;
};
// Wrap each handler with the logging middleware
export const POST = logRequestAndResponseDetails(handlers.POST);
export const GET = logRequestAndResponseDetails(handlers.GET);
export const DELETE = logRequestAndResponseDetails(handlers.DELETE);
export const PUT = logRequestAndResponseDetails(handlers.PUT);
```
```ts Remix route wrappers theme={null}
// app/routes/nile-api.ts
// In Remix, we need to add logging on both the loader and the action
import type { Route } from "./+types/home";
import { handlers } from "~/nile";
const { GET, POST, PUT, DELETE } = handlers;
export const loader = async ({ request }: Route.LoaderArgs) => {
// Log request headers
console.log('Request Headers:', JSON.stringify([...request.headers]));
let response;
switch (request.method.toUpperCase()) {
case "GET":
response = await GET(request);
break;
case "POST":
// Log request body for POST
const postBody = await request.text();
console.log('POST Request Body:', postBody);
response = await POST(request);
break;
case "PUT":
response = await PUT(request);
break;
case "DELETE":
response = await DELETE(request);
break;
default:
response = new Response("Method Not Allowed", { status: 405 });
}
// Log response status
console.log('Response Status:', response.status);
return response;
};
export const action = async ({ request }: Route.ActionArgs) => {
// Log request headers
console.log('Request Headers:', JSON.stringify([...request.headers]));
let response;
switch (request.method.toUpperCase()) {
case "POST":
// Log request body for POST
const postBody = await request.text();
console.log('POST Request Body:', postBody);
response = await POST(request);
break;
case "PUT":
response = await PUT(request);
break;
case "DELETE":
response = await DELETE(request);
break;
default:
response = new Response("Method Not Allowed", { status: 405 });
}
// Log response status
console.log('Response Status:', response.status);
return response;
};
```
```ts Express route wrappers theme={null}
// server.mjs
import 'dotenv/config';
import express from 'express';
import { Nile } from '@niledatabase/server';
import { NileExpressHandler } from '@niledatabase/server/express';
const startServer = async () => {
try {
const app = express();
const nile = Nile({
debug: true,
});
const { paths, handler } = await NileExpressHandler(nile);
console.log(paths);
app.use(express.json());
app.use(express.urlencoded({ extended: true }));
// Create logging wrapper
const withLogging = (handler) => async (req, res, next) => {
console.log('Request Headers:', req.headers);
if (req.method === 'POST') {
console.log('Request Body:', req.body);
}
return handler(req, res, next);
};
// Apply wrapper to specific routes
app.get(paths.get, withLogging(handler));
app.post(paths.post, withLogging(handler));
app.put(paths.put, withLogging(handler));
app.delete(paths.delete, withLogging(handler));
const PORT = process.env.PORT || 3040;
app.listen(PORT, () => {
console.log(`Server is running on port ${PORT}`);
});
} catch (error) {
console.error('Error starting server:', error);
process.exit(1);
}
};
startServer();
```
### Route overrides
In the examples below, we'll add a new route that will override the default route for `\auth\google\callback` with
custom logic. We are using `handlersWithContext` to get the `nile` object in the route handler.
`handlersWithContext` returns a tuple with the `nile` object, configured based on the response from the route handler,
and the response from the route handler.
```ts NextJS route overrides theme={null}
// app/api/auth/google/callback/route.ts
import { NextRequest } from "next/server";
// make sure you export handlersWithContext from nile
import { handlersWithContext } from "../../../[...nile]/nile";
import { registerTenants } from "@/lib/TenantRegistration";
export async function GET(req: NextRequest) {
// call the original route
const {nile, response} = await handlersWithContext.GET(req);
if (nile) {
const me = await nile.users.getSelf();
if ("id" in me) {
// custom logic is here
await registerTenants(me.id);
}
}
// return the original response from the route
return response;
}
```
```ts Remix route overrides theme={null}
// app/routes/auth-callback-google.tsx
// Also, add to routes.ts:
// route("api/auth/google/callback", "routes/auth-callback-google.tsx")
import type { Route } from "./+types/home";
import { handlers } from "~/nile";
const { GET } = handlers;
export const loader = async ({ request }: Route.LoaderArgs) => {
// Call Nile's GET handler
const nileResponse = await GET(request);
const setCookie = nileResponse.headers.get('set-cookie');
const hasSession = setCookie && setCookie.includes("nile.session-token");
if (hasSession) {
// If login was successful, register the tenants
const headers = new Headers();
headers.append('cookie', setCookie);
// Set the headers for the Nile API
const nileCtx = await nile.withContext({ headers });
// Fetch user information
const me = await nileCtx.users.getSelf();
if ("id" in me) {
// Custom logic: register tenants
await registerTenants(me.id);
}
}
return nileResponse;
};
```
```ts Express route overrides theme={null}
// server.mjs
// One big difference between express and other frameworks is that Nile's route handlers
// do not return a response by default. In order to use them in a custom handler,
// you need some extra configuration and extra handling of all routes
import 'dotenv/config';
import express from 'express';
import { Nile } from '@niledatabase/server';
import { NileExpressHandler } from '@niledatabase/server/express';
const startServer = async () => {
try {
const app = express();
const nile = Nile({
// debug: true,
});
// muteResponse is set to true to avoid sending the response back to the client from the route handler
// this is useful when you want to override the route with your own logic
const { paths, handler } = await NileExpressHandler(nile, {
muteResponse: true,
});
console.log(paths);
app.use(express.json());
app.use(express.urlencoded({ extended: true }));
// Add custom Google OAuth callback route - This has to be done before all the default routes
app.get('/auth/google/callback', async (req, res, next) => {
const { status, headers, body } = await handler(req, res, next);
if (status === 200) {
// Fetch user information
const nileCtx = nile.withContext({ headers });
const me = await nileCtx.users.getSelf();
if ('id' in me) {
// Custom logic: register tenants
await registerTenants(me.id);
}
}
res.status(status).set(headers);
if (typeof body === 'string') {
res.send(body);
} else {
res.json(body ?? {});
}
return;
});
// Create response logging wrapper
// This is critical to ensure that the response is sent back to the client from the route handlers
// So make sure you use with `{ muteResponse: true }` even if you don't want to log all the responses.
const withResponseLogging = (handler) => async (req, res, next) => {
try {
// Wait for the handler to complete
const result = await handler(req, res, next);
// Log the result
console.log('Handler result:', result);
const { status, headers, body } = result;
res.status(status).set(headers);
if (typeof body === 'string') {
res.send(body);
} else {
res.json(body ?? {});
}
return;
} catch (error) {
console.error('Handler error:', error);
throw error;
}
};
// Apply wrapper to routes
app.get(paths.get, withResponseLogging(handler));
app.post(paths.post, withResponseLogging(handler));
app.put(paths.put, withResponseLogging(handler));
app.delete(paths.delete, withResponseLogging(handler));
const PORT = process.env.PORT || 3040;
app.listen(PORT, () => {
console.log(`Server is running on port ${PORT}`);
});
} catch (error) {
console.error('Error starting server:', error);
process.exit(1);
}
};
startServer();
```
# Tenants
Source: https://thenile.dev/docs/auth/sdk-reference/javascript/server/tenants
Managing tenants for the current user
The Tenants module provides methods to create, fetch, update, delete, and interact with tenants that the currently authenticated user is a member of.
Each user may belong to multiple tenants, and most operations automatically default to the currently selected tenant. You can explicitly override this by providing a tenant ID to most methods.
Most methods assume the current user is authenticated. If not, a 401
Unauthorized error will be returned.
***
## Setting the Current Tenant
You can manually set the current tenant using `await nile.withContext({ tenantId })`. `withContext` returns a copy of the nile instance with a context that captures what was passed. It also accepts a second parameter for convenience for multiple requests.
```ts Returns a nile with the context theme={null}
const nileWithContext = await nile.withContext({
tenantId: '019612d7-56e7-7e87-8f30-ad6b05d85645',
});
```
```ts Self contained callback theme={null}
const [currentUser, listOfTenantUsers] = await nile.withContext(
{
tenantId: '019612d7-56e7-7e87-8f30-ad6b05d85645',
},
async (_nile) => Promise.all([_nile.users.getSelf, _nile.users.list]),
);
```
This method updates the internal configuration to use the given tenant in subsequent API calls.
Framework-specific integrations may set context automatically, so calling
`withContext` every time may not be necessary in those environments. see
[extensions](../../../concepts/extensions.mdx) for more information
This ID is typically sourced from:
* The result of `create()`
* A specific tenant via `get()`
* A selected tenant from `list()`
```ts setCurrentTenant from list theme={null}
const tenants = await nile.tenants.list();
if (!(tenants instanceof Response)) {
nile.withContext({ tenantId: tenants[0].id });
}
```
If not explicitly set, an error will be thrown by the SDK if the tenant Id is missing.
***
## create
Create a new tenant using `POST /api/tenants`. The current user is automatically linked to the new tenant.
```ts createTenant by name theme={null}
const tenant = await nile.tenants.create('Acme Inc');
```
```ts createTenant with ID theme={null}
const tenant = await nile.tenants.create({ name: 'My Org', id: 'custom-uuid' });
```
### Parameters
* `name`: Name of the tenant (required)
* `id`: Optional. Unique ID to use instead of auto-generating
### Returns
If successful, resolves to a `Tenant` object:
```ts theme={null}
interface Tenant {
id: string;
name: string;
}
```
Returns a `Response` with 401 or 400 on failure.
***
## get
Fetch a specific tenant or the current tenant using `GET /api/tenants/{id}`.
```ts getTenant by ID theme={null}
const tenant = await nile.tenants.get('0196...');
```
```ts getCurrentTenant theme={null}
const n = await nile.withContext({ tenantId: '0196...' });
const tenant = await n.tenants.get();
```
### Parameters
* `id`: Optional. If omitted, uses the current context
### Returns
A `Tenant` object or a raw `Response`.
***
## update
Update the name of the current tenant via `PUT /api/tenants/{id}`.
```ts theme={null}
const n = await nile.withContext({ tenantId: '0196...' });
const tenant = await n.tenants.update({ name: 'New Name' });
```
### Parameters
* `name`: New name of the tenant
* `id`: Optional. If omitted, uses context
### Returns
Updated `Tenant` object or a `Response`.
Only the name is currently updatable.
***
## delete
Mark a tenant as deleted using `DELETE /api/tenants/{id}`. This is a soft delete.
```ts delete by ID theme={null}
const res = await nile.tenants.delete('0196...');
```
```ts delete current theme={null}
const n = await nile.withContext({ tenantId: '0196...' });
await n.tenants.delete();
```
### Returns
A `Response` with status 204 on success.
***
## list
List all tenants the current user belongs to via `GET /api/tenants/by-user`.
```ts theme={null}
const tenants = await nile.tenants.list();
```
### Returns
Array of `Tenant` objects:
```ts theme={null}
[{ id: '...', name: '...' }];
```
401 if the user is unauthenticated.
***
## users
List users who are members of a given tenant (or the current one).
```ts theme={null}
const n = await nile.withContext({ tenantId: '0196...' });
const users = await n.tenants.users();
```
### Returns
Array of `User` objects, or a `Response` on error.
***
## addMember
Add a user to the current tenant using their user ID.
```ts theme={null}
const n = await nile.withContext({ tenantId: '0196...' });
await n.tenants.addMember('user-id');
```
### Returns
The added `User` object or a `Response`.
***
## removeMember
Remove a user from the tenant.
```ts theme={null}
const n = await nile.withContext({ tenantId: '0196...' });
await n.tenants.removeMember('user-id');
```
### Returns
A `Response`.
***
## leaveTenant
Removes the current user from a tenant they belong to.
```ts theme={null}
const n = await nile.WithContext({ tenantId: '0196...' });
await n.tenants.leaveTenant();
```
### Returns
A `Response`.
***
## invites
Fetch all invites for the current tenant:
```ts theme={null}
const n = await nile.withContext({ tenantId: '0196...' });
const invites = await n.tenants.invites();
```
### Returns
An array of `Invite` objects.
***
## invite
Invite a new user by email to the current tenant. the `callbackUrl` is the value that is used to return the user to a page that renders HTML. There is a primary endpoint that is used to exchange the token in the email to join the tenant. Upon a successful exchange, the user will be redirected to the `callbackUrl`
```ts invite theme={null}
const n = await nile.withContext({ tenantId: '0196...' });
await n.tenants.invite('user@example.com');
```
```ts invite with callback theme={null}
const n = await nile.withContext({ tenantId: '0196...' });
await n.tenants.invite({
email: 'user@example.com',
callbackUrl: 'https://myapp.com/welcome',
});
```
### Parameters
* `email`: Email of the user to invite
* `callbackUrl`: Optional URL to redirect after acceptance
* `redirectUrl`: Optional URL used as the base in the email
### Returns
An `Invite` object or a `Response`.
***
## acceptInvite
Accept an invite using an emailed token and email address.
```ts theme={null}
await nile.tenants.acceptInvite({ identifier: 'email', token: '123' });
```
### Returns
A `Response` indicating success or failure.
***
## deleteInvite
Delete a previously sent invite.
```ts theme={null}
await nile.tenants.deleteInvite('invite-id');
```
### Returns
A `Response`.
# Users
Source: https://thenile.dev/docs/auth/sdk-reference/javascript/server/users
Managing the currently authenticated user
The Users module provides authenticated users with a focused set of methods to retrieve, update, verify, or delete their own profile information.
These APIs are scoped to the logged-in user. User creation, session management, and tenant access are now delegated to the `auth` and `tenants` modules.
This module only exposes endpoints related to the current user (`/api/me`) and
does not support administrative actions like creating or managing other users.
Creating users would be done by the user via `nile.auth.signUp()`, and
managing users that are not yourself is not yet support.
***
## getSelf
The `nile.users.getSelf()` method retrieves the profile of the currently authenticated user. Internally, this maps to a `GET /api/me` request.
This is typically used to display account settings, personalize the UI, or check tenant memberships after a login.
```ts getSelf theme={null}
const me = await nile.users.getSelf();
```
```ts getSelf raw theme={null}
const res = await nile.users.getSelf(true); // raw Response
```
### Returns
A `User` object or raw `Response`:
```ts theme={null}
interface User {
id: string;
email: string;
name?: string;
familyName?: string;
givenName?: string;
picture?: string;
created: string;
updated?: string;
emailVerified?: boolean;
tenants: { id: string }[];
}
```
If the user is not authenticated or their session is invalid, this will return a 401.
***
## updateSelf
The `nile.users.updateSelf()` method allows the currently authenticated user to update their own profile using `PUT /api/me`.
Use this to let users edit profile settings like name or profile picture. Fields not included in the update are left unchanged.
```ts updateSelf theme={null}
await nile.users.updateSelf({ name: 'Jane Doe', picture: 'https://example.com/photo.png' });
```
```ts updateSelf raw theme={null}
await nile.users.updateSelf({ name: 'Jane Doe' }, true); // returns raw Response
```
### Allowed Fields
* `name`: Full display name of the user
* `familyName`: Surname or last name
* `givenName`: First name
* `picture`: Optional URL to an avatar or profile image
* `emailVerified`: The date when the email was verified
The following fields cannot be modified: `email`, `tenants`, `created`,
`updated`, or `id`.
If the user is not authenticated, a 401 will be returned.
***
## removeSelf
This method deletes the current user's account by sending a `DELETE /api/me` request.
This is a soft delete operation — the user will no longer be able to sign in, but their historical data may still exist in the system for audit purposes. This action also clears all authentication headers that are maintained server-side. It would be necessary remove client side cookies as well for completeness.
```ts theme={null}
await nile.users.removeSelf();
```
### Returns
A `Response` object:
* `200 OK` if the user was successfully marked for deletion
* `401 Unauthorized` if not logged in
* `404 Not Found` if the user does not exist
This is often used in account settings for "Delete My Account" functionality.
***
## verifySelf
This method initiates an email verification flow for the current user by POSTing to `/auth/verify-email`.
In production, this sends an email containing a link to verify the account, if configured. In development or testing, it can optionally skip the email step and mark the user as verified.
```ts verifySelf theme={null}
await nile.users.verifySelf({ callbackUrl: 'https://example.com/verified' });
```
```ts verifySelf bypass theme={null}
await nile.users.verifySelf({ bypassEmail: true });
```
```ts verifySelf raw theme={null}
await nile.users.verifySelf(true); // bypass + raw response
```
### Options
* `callbackUrl`: Optional. Where to redirect the user after successful verification.
* `bypassEmail`: Optional. If `true`, skips the email and sets `emailVerified = true` directly.
This bypass is useful for local development and CI environments where SMTP is
not configured.
### Returns
* If `bypassEmail` is used, resolves to the updated `User` object.
* Otherwise, resolves to a `Response` from the verification endpoint.
Use `nile.auth.signUp()` and `nile.auth.signIn()` for authentication, and
manage other users through tenant-related APIs where applicable.
# Self Host Nile Auth
Source: https://thenile.dev/docs/auth/selfhosting
Learn how to self-host Nile Auth in your own infrastructure
Nile provides a cloud offering to help build multi-tenant apps. You can also get started with Nile's Docker image
and try Nile locally. [Join our discord](https://discord.com/invite/8UuBB84tTy) to give feedback or ask questions about running Nile locally.
* [Docker](https://www.docker.com/get-started)
* Postgres client. We'll use `psql` in this guide.
```bash theme={null}
docker run -p 5432:5432 -p 3000:3000 -ti ghcr.io/niledatabase/testingcontainer:latest
```
This will start a Postgres database with Nile extensions installed. It will also start Nile Auth (optional).
If this is the first time you are running the container, it will also pull the latest image,create the `test` database
and the `00000000-0000-0000-0000-000000000000` user.
You can use `psql` with the following connection string:
```bash theme={null}
psql postgres://00000000-0000-0000-0000-000000000000:password@localhost:5432/test
```
Or, if you are using a different client, you use the following connection details:
```
Host: localhost
Port: 5432
Database: test
Username: 00000000-0000-0000-0000-000000000000
Password: password
```
Nile Auth service is running and listening on port 3000. We will do a curl to check if you can reach it and if it is returning the right information.
The curl command queries the list of providers. By default, it should only have the email/credential option.
```bash theme={null}
curl http://localhost:3000/v2/databases/01000000-0000-7000-8000-000000000000/auth/providers
```
Output:
```bash theme={null}
{"credentials":{"id":"credentials","name":"credentials","type":"credentials","signinUrl":"https://null/api/auth/signin/credentials","callbackUrl":"https://null/api/auth/callback/credentials"}}%
```
This guide uses Next.js with App Router, Typescript and Tailwind CSS. If you have a different framework in mind, you can find additional guides under "Frameworks"
in the sidebar. Initialize a new Next.js project with the following command and give it a name:
```bash theme={null}
npx create-next-app@latest nile-app --yes
```
Create a .env file in the next app folder that you just created and paste the values below. This will make the Next app that we just created use the local Nile Postgres and Nile Auth.
```bash theme={null}
NILEDB_USER=00000000-0000-0000-0000-000000000000
NILEDB_PASSWORD=password
NILEDB_API_URL=http://localhost:3000/v2/databases/01000000-0000-7000-8000-000000000000
NILEDB_POSTGRES_URL=postgres://localhost:5432/test
```
```bash theme={null}
npm install @niledatabase/server @niledatabase/react
@niledatabase/client @niledatabase/nextjs
```
Your application must expose API routes to handle authentication operations.
Create a folder called `api` under the `app` folder and a folder called `[...nile]` under it:
```bash theme={null}
mkdir -p app/api/\[...nile\]
```
Create following files handle the calls to your server, as well as expose the `nile` instance to your application:
```typescript app/api/[...nile]/nile.ts theme={null}
import { Nile } from '@niledatabase/server';
import { nextJs } from '@niledatabase/nextjs';
export const nile = Nile({ extensions: [nextJs] });
```
```typescript app/api/[...nile]/route.ts theme={null}
import { nile } from './nile';
export const { POST, GET, DELETE, PUT } = nile.handlers;
```
Your application will interact with above authentication routes using SDK components. Replace the boilerplate `app/page.tsx` with the following:
`/app/page.jsx`
```jsx theme={null}
import {
SignOutButton,
SignUpForm,
SignedIn,
SignedOut,
TenantSelector,
UserInfo,
} from "@niledatabase/react";
import "@niledatabase/react/styles.css";
export default function SignUpPage() {
return (
);
}
```
```bash theme={null}
npm run dev
```
Navigate to localhost to see the page. You should see a signup form that looks like this:
Enter a dummy email and password into the SignUpForm.
If all went well, you will be logged in automatically and see the user profile and an organization switcher that allows you to create new organizations and switch between them:
* [Explore Frontend Frameworks](/auth/quickstarts/nextjs)
* [Learn About Backend Integration](/auth/quickstarts/express)
# Discord
Source: https://thenile.dev/docs/auth/singlesignon/discord
Integrate Discord Single Sign-On with Nile Auth
1. Create a [Discord Developer](https://discord.com/developers/applications) account.
2. Follow the documentation for [creating an OAuth2 application](https://discord.com/developers/docs/topics/oauth2) in Discord.
3. Obtain a **Client ID** and **Client Secret** from [Discord OAuth](https://discord.com/developers/applications). Be sure to set the redirect URL to your application handling the Nile Auth requests.
Save them to your database at `console.thenile.dev` under Discord in **Tenants
& Users -> Configuration**
The button will redirect the user to Discord for authentication. Upon successful authentication, the user will be redirected back to your application.
```jsx theme={null}
import { DiscordSignInButton } from '@niledatabase/react';
function App() {
return (
);
}
```
## Related Topics
* [OAuth Concepts](/auth/concepts/oauth)
* [User Management](/auth/concepts/users)
* [Other SSO Providers](/auth/singlesignon/github)
# GitHub
Source: https://thenile.dev/docs/auth/singlesignon/github
Integrate GitHub Single Sign-On with Nile Auth
1. Create a [GitHub Developer](https://github.com/settings/developers) account.
2. Follow the documentation for [creating an OAuth App](https://docs.github.com/en/developers/apps/building-oauth-apps/creating-an-oauth-app) in GitHub.
3. Obtain a **Client ID** and **Client Secret** from [GitHub OAuth](https://github.com/settings/developers). Be sure to set the redirect URL to your application handling the Nile Auth requests.
Save them to your database at `console.thenile.dev` under GitHub in **Tenants & Users -> Configuration**\\
The button will redirect the user to GitHub for authentication. Upon successful authentication, the user will be redirected back to your application.
```jsx theme={null}
import { GitHub } from '@niledatabase/react';
function App() {
return (
);
}
```
## Related Topics
* [OAuth Concepts](/auth/concepts/oauth)
* [User Management](/auth/concepts/users)
* [Other SSO Providers](/auth/singlesignon/google)
# Google
Source: https://thenile.dev/docs/auth/singlesignon/google
Integrate Google Single Sign-On with Nile Auth
1. Create a [Google Developer](https://console.developers.google.com/) account.
2. Follow the documentation for [creating an OAuth 2.0 Client ID](https://developers.google.com/identity/protocols/oauth2) in Google.
3. Obtain a **Client ID** and **Client Secret** from [Google OAuth](https://console.cloud.google.com/apis/credentials).
Be sure to set the redirect URL to your application handling the Nile Auth requests.
If you are running locally and used the routes generated by `@niledatabase/server`, the redirect URL is `http://localhost:3000/api/auth/callback/google`.
Save them to your database at `console.thenile.dev` under Google in **Tenants
& Users -> Configuration**
The button will redirect the user to Google for authentication. Upon successful authentication, the user will be redirected back to your application.
```jsx theme={null}
import { Google } from '@niledatabase/react';
function App() {
return (
);
}
```
## Related Providers
* [GitHub SSO](/auth/singlesignon/github)
* [Microsoft SSO](/auth/singlesignon/microsoft)
* [Discord SSO](/auth/singlesignon/discord)
# HubSpot
Source: https://thenile.dev/docs/auth/singlesignon/hubspot
Integrate HubSpot Single Sign-On with Nile Auth
1. Create a [HubSpot Developer](https://developers.hubspot.com/) account
2. Follow the documentation for [creating an application](https://developers.hubspot.com/docs/guides/apps/public-apps/overview) in HubSpot.
3. Obtain a **Client ID** and **Client Secret** from [HubSpot OAuth](https://developers.hubspot.com/docs/guides/apps/authentication/working-with-oauth). Be sure to set the redirect URL to your application handling the Nile Auth requests.
Save them to your database at `console.thenile.dev` under HubSpot in **Tenants
& Users -> Configuration**
The button will redirect the user to HubSpot for authentication. Upon successful authentication, the user will be redirected back to your application.
```jsx theme={null}
import { HubSpot } from '@niledatabase/react';
function App() {
return (
); }
```
## Related Topics
* [OAuth Concepts](/auth/concepts/oauth)
* [User Management](/auth/concepts/users)
* [Other SSO Providers](/auth/singlesignon/google)
```
```
# LinkedIn
Source: https://thenile.dev/docs/auth/singlesignon/linkedin
Integrate LinkedIn Single Sign-On with Nile Auth
1. Create a [LinkedIn Developer](https://www.linkedin.com/developers/) account.
2. Navigate to the **My Apps** section and create a new application.
3. Obtain a **Client ID** and **Client Secret** from the [LinkedIn OAuth settings](https://www.linkedin.com/developers/apps/). Be sure to set the redirect URL to your application handling the Nile Auth requests.
Save the **Client ID** and **Client Secret** to your database at `console.thenile.dev` under LinkedIn in **Tenants & Users -> Configuration**\\
The button will redirect the user to LinkedIn for authentication. Upon successful authentication, the user will be redirected back to your application.
```jsx theme={null}
import { LinkedIn } from '@niledatabase/react';
function App() {
return (
);
}
```
## Related Topics
* [OAuth Concepts](/auth/concepts/oauth)
* [User Management](/auth/concepts/users)
* [Other SSO Providers](/auth/singlesignon/github)
# Microsoft Azure
Source: https://thenile.dev/docs/auth/singlesignon/microsoft
Integrate Azure Single Sign-On with Nile Auth
1. Create an [Azure AD](https://portal.azure.com/) account if you don’t have one.
2. Navigate to the [Azure Portal](https://portal.azure.com/) and go to **Azure Active Directory** > **App registrations**.
3. Click **New registration**, enter a name for your app, and set the redirect URI to your application handling the Nile Auth requests.
4. Obtain the **Application (client) ID**, **Directory (tenant) ID**, and create a **Client Secret** under **Certificates & secrets**.
Save the **Client ID**, **Client Secret**, and **Tenant ID** to your database at `console.thenile.dev` under Azure in **Tenants & Users -> Configuration**\\
The button will redirect the user to Azure AD for authentication. Upon successful authentication, the user will be redirected back to your application.
```jsx theme={null}
import { AzureSignInButton } from '@niledatabase/react';
function App() {
return (
);
}
```
## Related Topics
* [OAuth Concepts](/auth/concepts/oauth)
* [User Management](/auth/concepts/users)
* [Other SSO Providers](/auth/singlesignon/google)
# Slack
Source: https://thenile.dev/docs/auth/singlesignon/slack
Integrate Slack Single Sign-On with Nile Auth
1. Create a [Slack Developer](https://api.slack.com/apps) account.
2. Navigate to the **Your Apps** section and create a new application.
3. Obtain a **Client ID** and **Client Secret** from the [Slack OAuth settings](https://api.slack.com/apps). Be sure to set the redirect URL to your application handling the Nile Auth requests.
Save the **Client ID** and **Client Secret** to your database at `console.thenile.dev` under Slack in **Tenants & Users -> Configuration**\\
The button will redirect the user to Slack for authentication. Upon successful authentication, the user will be redirected back to your application.
```jsx theme={null}
import { Slack } from '@niledatabase/react';
function App() {
return (
);
}
```
## Related Topics
* [OAuth Concepts](/auth/concepts/oauth)
* [User Management](/auth/concepts/users)
* [Other SSO Providers](/auth/singlesignon/google)
# X (Twitter)
Source: https://thenile.dev/docs/auth/singlesignon/x
Integrate X (Twitter) Single Sign-On with Nile Auth
1. Create a [X Developer](https://developer.x.com/) account.
2. Navigate to the **Developer Portal** and create a new application.
3. Obtain a **Client ID** and **Client Secret** from the [X OAuth settings](https://developer.x.com/). Be sure to set the redirect URL to your application handling the Nile Auth requests.
Save the **Client ID** and **Client Secret** to your database at `console.thenile.dev` under X in **Tenants & Users -> Configuration**\\
The button will redirect the user to X for authentication. Upon successful authentication, the user will be redirected back to your application.
```jsx theme={null}
import { X } from '@niledatabase/react';
function App() {
return (
);
}
```
## Related Topics
* [OAuth Concepts](/auth/concepts/oauth)
* [User Management](/auth/concepts/users)
* [Other SSO Providers](/auth/singlesignon/google)
# Connecting to Nile
Source: https://thenile.dev/docs/cli/connect_nile
Nile CLI supports two primary methods of authentication:
1. Browser-based OAuth authentication (Recommended)
2. API key authentication
## Browser-based Authentication
Browser-based authentication provides a secure and convenient way to connect to Nile. When you run the login command, the CLI will:
1. Open your default browser
2. Direct you to Nile's authentication page
3. After successful login, automatically configure your CLI
```bash Browser Login theme={null}
# Start the authentication flow
nile connect login
# Example output
Opening authorization URL: https://console.thenile.dev/authorize
Authentication successful!
✓ Successfully connected to Nile!
```
### Authentication Flow
The following illustrate the browser-based authentication flow:
1. CLI initiates auth flow
2. Browser opens to Nile login
3. User authenticates
4. Token returned to CLI
5. CLI stores credentials
### Verifying Connection
After authentication, verify your connection:
```bash Verify Connection theme={null}
# Check connection status
nile connect status
# Example output
✓ Connected to Nile
# Try a simple command
nile workspace list
```
### Session Management
Browser-based authentication creates a session that:
* Remains valid for extended periods
* Automatically refreshes when needed
* Can be explicitly terminated
```bash Session Management theme={null}
# Check current session
nile connect status
# End your session
nile connect logout
# Start a new session
nile connect login
```
## API Key Authentication
For automated workflows or CI/CD environments, you can use API key authentication. API keys can be:
* Used directly in commands
* Stored in configuration
* Set via environment variables
### Using API Keys
```bash API Key Examples theme={null}
# 1. Direct usage in commands
nile --api-key YOUR_API_KEY db list
# 2. Save in configuration
nile config --api-key YOUR_API_KEY
# 3. Use environment variable
export NILE_API_KEY=YOUR_API_KEY
nile db list
```
### API Key Best Practices
```bash API Key Security theme={null}
# 1. Store in environment file
echo "NILE_API_KEY=your-api-key" >> .env
echo ".env" >> .gitignore
# 2. Use in CI/CD (GitHub Actions example)
env:
NILE_API_KEY: ${{ secrets.NILE_API_KEY }}
# 3. Regular rotation
nile config --api-key NEW_API_KEY
```
## Common Issues and Debugging
### Authentication Failures
When authentication fails:
```bash Auth Issues theme={null}
# 1. Connection timeout
nile connect login
Error: Authentication timed out after 2 minutes
# Solution: Try again or check network
# 2. Invalid API key
nile --api-key INVALID_KEY db list
Error: Invalid API key
# Solution: Verify API key or try browser auth
nile connect login
```
### Network Issues
When experiencing network problems:
```bash Network Debug theme={null}
# Enable debug output
nile --debug connect login
# Example output
Debug - Auth parameters:
Debug - Auth URL: https://console.thenile.dev/authorize
Debug - Starting auth with config:
Debug - Received callback request: /callback
```
### Session Issues
When encountering session problems:
```bash Session Issues [expandable] theme={null}
# 1. Check session status
nile connect status
Not connected to Nile
# 2. Clear existing session
nile connect logout
# 3. Remove stored credentials
rm ~/.nile/credentials
# 4. Start fresh session
nile connect login
```
## Environment Configuration
Configure your environment for different authentication methods:
```bash Environment Setup theme={null}
# 1. Development (browser-based)
nile connect login
nile config --workspace dev
# 2. CI/CD (API key)
export NILE_API_KEY=your-api-key
export NILE_WORKSPACE=ci-cd
# 3. Production (API key)
export NILE_API_KEY=prod-api-key
export NILE_WORKSPACE=production
```
## Workspace Selection
After authentication, select your workspace:
```bash Workspace Selection theme={null}
# List available workspaces
nile workspace list
# Set default workspace
nile config --workspace my-workspace
# Verify workspace
nile workspace show
```
## Related Commands
* `nile connect status` - Check connection status
* `nile connect logout` - End current session
* `nile config` - Configure CLI settings
* `nile workspace list` - List available workspaces
# Global Flags
Source: https://thenile.dev/docs/cli/global-flags
Command line flags that can be used with any Nile CLI command
Global flags can be used with any command in the Nile CLI. They provide consistent functionality across all commands and help customize the CLI's behavior.
## Authentication Flags
Flags for authenticating with Nile's services.
```bash Authentication Examples theme={null}
# Using API key
nile --api-key YOUR_API_KEY db list
# Using workspace
nile --workspace my-workspace db list
# Using database
nile --db my-database db list
# Combining flags
nile --api-key YOUR_API_KEY --workspace my-workspace db list
```
| Flag | Description | Default |
| -------------------- | -------------------------- | -------------------------------------- |
| `--api-key ` | API key for authentication | Environment variable: `NILE_API_KEY` |
| `--workspace ` | Workspace to use | Environment variable: `NILE_WORKSPACE` |
| `--db ` | Database to use | Environment variable: `NILE_DB` |
## Output Format Flags
Customize how command output is displayed.
```bash Format Examples {2-4} theme={null}
# JSON output
nile --format json db list
# CSV output
nile --format csv db list
# Disable colors
nile --no-color db list
```
| Flag | Description | Default |
| ----------------- | -------------------------------- | ------- |
| `--format ` | Output format (human, json, csv) | `human` |
| `--color` | Enable colored output | `true` |
| `--no-color` | Disable colored output | - |
## Host Configuration Flags
Configure connection endpoints for Nile services.
```bash Host Configuration theme={null}
# Custom database host
nile --db-host custom.db.example.com db list
# Custom global host
nile --global-host api.example.com db list
# Custom auth URL
nile --auth-url auth.example.com connect login
```
| Flag | Description | Default |
| ---------------------- | ------------------------- | ---------------------------------------- |
| `--db-host ` | Custom database host | Environment variable: `NILE_DB_HOST` |
| `--global-host ` | Custom global host | Environment variable: `NILE_GLOBAL_HOST` |
| `--auth-url ` | Custom authentication URL | Environment variable: `NILE_AUTH_URL` |
## Debug Flag
Enable detailed output for troubleshooting.
```bash Debug Examples theme={null}
# Enable debug output
nile --debug db list
# Debug with other flags
nile --debug --format json db list
# Debug authentication
nile --debug connect login
```
| Flag | Description | Default |
| --------- | ------------------- | ------- |
| `--debug` | Enable debug output | `false` |
## Environment Variables
All global flags can also be set using environment variables:
```bash Environment Variables [expandable] theme={null}
# Authentication
export NILE_API_KEY=your-api-key
export NILE_WORKSPACE=your-workspace
export NILE_DB=your-database
# Host Configuration
export NILE_DB_HOST=your-db-host
export NILE_GLOBAL_HOST=your-global-host
export NILE_AUTH_URL=your-auth-url
# Debug Mode
export NILE_DEBUG=true
```
## Flag Precedence
When the same setting is specified in multiple ways, the following precedence order is used (highest to lowest):
1. Command line flags
2. Environment variables
3. Configuration file values
4. Default values
```bash Precedence Examples theme={null}
# Flag overrides environment variable
export NILE_WORKSPACE=workspace1
nile --workspace workspace2 db list # Uses workspace2
# Flag overrides config
nile config --workspace workspace1
nile --workspace workspace2 db list # Uses workspace2
```
# Interacting with Database
Source: https://thenile.dev/docs/cli/interacting_db
The Nile CLI provides a powerful set of commands for managing your databases. With the CLI, you can create, delete, and manage databases, connect to them using standard PostgreSQL tools, and more.
```bash theme={null}
nile db [command] [options]
```
Available commands:
* `list` - View all databases
* `show` - Display database details
* `create` - Create a new database
* `delete` - Remove a database
* `regions` - List available regions
* `psql` - Connect using PostgreSQL CLI
* `connectionstring` - Get connection details
## Setting Workspace
Before using the `nile db` command, you can set the workspace to operate in.
```bash theme={null}
nile config --workspace
```
### Example
```bash Workspace Example theme={null}
# Set workspace to 'test'
nile config --workspace test
```
The workspace can also be set using the `NILE_WORKSPACE` environment variable
or by using the flag `--workspace` in the command.
## Listing Databases
The `list` command shows all databases in your workspace.
```bash theme={null}
nile db list [options]
```
### Options
| Flag | Description | Default |
| ------------- | -------------------------------- | ----------------- |
| `--format` | Output format (human, json, csv) | human |
| `--workspace` | Specific workspace to list from | Current workspace |
### Examples
```bash List Examples theme={null}
# Basic list
nile db list
# JSON format
nile db list --format json
# Specific workspace
nile db list --workspace test
```
## Setting active database in config
You can set the active database in the config file. This simplifies other db commands. Youc an also override the db name using the flag `--database` in the command.
```bash theme={null}
nile config --database
```
### Example
```bash theme={null}
nile config --database determined_house
```
## Showing Database Details
The `show` command provides detailed information about a specific database.
```bash theme={null}
nile db show [databaseName] [options]
```
### Options
| Flag | Description | Default |
| ------------- | --------------------------------- | ----------------- |
| `--format` | Output format (human, json, csv) | human |
| `--workspace` | Workspace containing the database | Current workspace |
### Examples
```bash Show Examples theme={null}
# Show specific database
nile db show myapp-db
# Show current database
nile db show
# Detailed JSON output
nile db show myapp-db --format json
```
## Listing Regions
The `regions` command shows available regions for database creation.
```bash theme={null}
nile db regions [options]
```
### Options
| Flag | Description | Default |
| ------------- | -------------------------------- | ----------------- |
| `--format` | Output format (human, json, csv) | human |
| `--workspace` | Workspace to list regions for | Current workspace |
### Examples
```bash Region Examples theme={null}
# List all regions
nile db regions
# JSON format
nile db regions --format json
# Check regions in specific workspace
nile db regions --workspace test
```
## Creating Databases
The `create` command sets up a new database in your workspace.
```bash theme={null}
nile db create [options]
```
### Required Options
| Flag | Description |
| ---------- | -------------------------------- |
| `--name` | Name for the new database |
| `--region` | Region to create the database in |
### Additional Options
| Flag | Description | Default |
| ------------- | -------------------------------- | ----------------- |
| `--format` | Output format (human, json, csv) | human |
| `--workspace` | Workspace to create in | Current workspace |
### Examples
```bash Create Examples theme={null}
# Create new database
nile db create --name myapp_db --region AWS_US_WEST_2
# Create in specific workspace
nile db create --name prod_db --region AWS_US_WEST_2 --workspace test
```
Use the nile db region command to see the available regions.
After creating the db, you can use the nile db show command to see the database details. You also want to set the config to the db you just created.
## Deleting Databases
The `delete` command removes a database permanently.
```bash theme={null}
nile db delete [databaseName] [options]
```
### Options
| Flag | Description | Default |
| ------------- | --------------------------------- | ----------------- |
| `--force` | Skip confirmation prompt | false |
| `--workspace` | Workspace containing the database | Current workspace |
### Examples
```bash Delete Examples theme={null}
# Delete with confirmation
nile db delete myapp_db
# Force delete without confirmation
nile db delete myapp_db --force
# Delete from specific workspace
nile db delete myapp_db --workspace development
```
## Using PostgreSQL CLI
The `psql` command opens an interactive PostgreSQL terminal.
```bash theme={null}
nile db psql [databaseName] [options]
```
### Options
| Flag | Description | Default |
| ------------- | --------------------------------- | ----------------- |
| `--workspace` | Workspace containing the database | Current workspace |
### Examples
```bash PSQL Examples theme={null}
# Connect to database using the config
nile db psql
# Connect in specific db and workspace
nile db psql --name determined_house --workspace test
```
## Getting Connection Strings
The `connectionstring` command provides database connection details. Currently, it only supports the psql format.
```bash theme={null}
nile db connectionstring --psql
```
### Examples
```bash Connection String Examples theme={null}
# Get basic connection string
nile db connectionstring --psql
```
## Importing Data
The `copy` command allows you to import data from files into database tables. The target table must already exist in the database with the appropriate columns that match your input data.
```bash theme={null}
nile db copy [options]
```
### Required Options
| Flag | Description |
| -------------- | ------------------------------------------------- |
| `--table-name` | Name of the target table (must exist in database) |
| `--format` | File format (csv or text) |
| `--file-name` | Path to the input file |
### Additional Options
| Flag | Description | Default |
| --------------- | ------------------------------------ | --------------------- |
| `--delimiter` | Column delimiter character | comma for CSV |
| `--column-list` | Comma-separated list of column names | All columns from file |
| `--debug` | Show detailed progress information | false |
### Examples
```bash Copy Examples theme={null}
# Basic CSV import
nile db copy --table-name products --format csv --file-name products.csv
# Import with specific columns
nile db copy --table-name products --format csv --file-name products.csv --column-list "id,name,price"
# Import tab-delimited text file
nile db copy --table-name products --format text --file-name products.txt --delimiter "\t"
# Import with progress details
nile db copy --table-name products --format csv --file-name products.csv --debug
```
### Features
* Shows progress bar with completion status and import speed
* Supports CSV files with headers
* Allows custom column mapping
* Provides detailed progress information in debug mode
* Processes data in batches for optimal performance
Make sure your target table exists and has the correct schema before running
the import. The columns in your input file should match the table's structure.
## Common Workflows
### Setting Up a New Database
```bash New Database Workflow theme={null}
# 1. List available regions
nile db regions
# 2. Create database
nile db create --name myapp_db --region AWS_US_WEST_2
# 3. Verify creation
nile db show myapp_db
# 4. Connect to database
nile db psql myapp_db
```
### Managing Multiple Environments
```bash Environment Management theme={null}
# Development setup
nile db create --name dev_db --region AWS_US_WEST_2 --workspace development
# Production setup
nile db create --name prod_db --region AWS_US_EAST_1 --workspace production
# Get connection strings
nile db connectionstring dev_db --workspace development
nile db connectionstring prod_db --workspace production
```
### Database Cleanup
```bash Cleanup Workflow theme={null}
# 1. List all databases
nile db list
# 2. Check database details
nile db show old_db
# 3. Delete unused database
nile db delete old_db --force
# 4. Verify deletion
nile db list
```
## Getting Help
For detailed help on any command:
```bash theme={null}
# General database help
nile db help
# Specific command help
nile db help create
nile db help psql
```
## Related Topics
* [Connecting to Nile](./connect_nile) - Authentication guide
* [Working with Tenants](./managing_tenants) - Tenant management
* [Configuration](./setting_configs) - CLI configuration
* [Workspaces](./managing_workspaces) - Workspace management
# Introduction
Source: https://thenile.dev/docs/cli/introduction
Command line interface for managing Nile databases
## Installation
```bash Terminal Installation theme={null}
# Using npm (recommended)
npm install -g niledatabase
# Using yarn
yarn global add niledatabase
# Using bun
bun install -g niledatabase
```
## Getting Started with Nile CLI
Nile CLI is a powerful command-line tool that helps you manage your Nile databases directly from your terminal. With Nile CLI, you can:
* Create and manage databases across multiple regions
* Handle multi-tenant applications with built-in tenant management
* Connect to your databases using standard PostgreSQL tools
* Manage workspaces and configurations
The code is available on [Github](https://github.com/niledatabase/cli). You are welcome to contribute to the project.
```bash Quick Start theme={null}
# Verify installation
nile --version
# Login to Nile
nile connect login
# List your databases
nile db list
```
## Authentication
There are three ways to authenticate with Nile:
1. **Browser-based Authentication** (Recommended)
2. **API Key Authentication**
3. **Environment Variables**
```bash Authentication Methods [expandable] theme={null}
# 1. Browser-based Authentication
nile connect login
# 2. API Key Authentication
nile --api-key YOUR_API_KEY db list
# or save it in config
nile config --api-key YOUR_API_KEY
# 3. Environment Variables
export NILE_API_KEY=YOUR_API_KEY
nile db list
```
## Basic Configuration
Before using the CLI, you should set up your workspace and default database:
```bash Basic Setup {2-3} theme={null}
# Set your workspace
nile config --workspace your-workspace
# Set your default database
nile config --db your-database
# Verify your configuration
nile config
```
## Output Formats
The CLI supports multiple output formats for easy integration with other tools:
```bash Output Examples theme={null}
# Human-readable format (default)
nile db list
# JSON format
nile --format json db list
# CSV format
nile --format csv db list
```
## Platform Requirements
The CLI requires:
* Node.js v16 or later
* npm, yarn, or bun package manager
* PostgreSQL command line tools (for database connections)
```bash Version Check Commands theme={null}
# Check Node.js version
node --version
# Check npm version
npm --version
# Check PostgreSQL tools
psql --version
```
## Common Workflows
Here are some common workflows to get you started:
### Setting Up a New Database
```bash Database Setup Example {3} theme={null}
# List available regions
nile db regions
# Create a new database
nile db create --name mydb --region AWS_US_WEST_2
# Connect to your database
nile db psql --name mydb
```
### Managing Tenants
```bash Tenant Management Examples [expandable] theme={null}
# Create a new tenant
nile tenants create --name "My Tenant"
# List all tenants
nile tenants list
# Update a tenant
nile tenants update --id tenant-123 --new_name "Updated Name"
# Delete a tenant
nile tenants delete --id tenant-123
# Show tenant details
nile tenants show --id tenant-123
```
## Getting Help
The CLI includes comprehensive help for all commands:
```bash Help Examples theme={null}
# Show general help
nile --help
# Show help for specific command
nile db --help
# Show help for subcommand
nile tenants create --help
```
## Debugging
When things go wrong, use the debug flag for detailed output:
```bash Debug Commands {2,5} theme={null}
# Enable debug output
nile --debug db list
# Check connection status
nile connect status --debug
# View configuration
nile config
```
# Local Development
Source: https://thenile.dev/docs/cli/local_development
The `nile local` command helps you manage a local development environment using Docker. This environment provides a PostgreSQL database pre-configured with Nile's schema and features.
## Overview
```bash theme={null}
nile local [command] [options]
```
Available commands:
* `start` - Start the local environment
* `stop` - Stop the local environment
* `info` - Display connection information
## Prerequisites
Before using the local environment, ensure you have:
1. Docker installed and running
2. PostgreSQL client tools (psql) installed
3. Sufficient permissions to run Docker commands
```bash Prerequisites Check theme={null}
# Check Docker installation
docker --version
# Check PostgreSQL tools
psql --version
```
## Starting the Environment
The `start` command launches a Docker container with a pre-configured PostgreSQL database.
```bash theme={null}
nile local start [options]
```
### Options
| Flag | Description | Default |
| ------------- | ------------------------------------------- | ------- |
| `--no-prompt` | Start without prompting for psql connection | false |
### Examples
```bash Start Examples theme={null}
# Start with interactive prompt
nile local start
# Start without psql prompt
nile local start --no-prompt
```
### Sample Output
```bash Start Output theme={null}
✓ Latest Nile testing container pulled successfully
✓ Database is ready
✓ Local development environment started successfully
Connection Information:
Host: localhost
Port: 5432
Database: test
Username: 00000000-0000-0000-0000-000000000000
Password: password
Would you like to connect using psql? (y/N)
```
## Stopping the Environment
The `stop` command gracefully stops and removes the Docker container.
```bash theme={null}
nile local stop
```
### Examples
```bash Stop Examples theme={null}
# Stop the environment
nile local stop
```
### Sample Output
```bash Stop Output theme={null}
✓ Local environment stopped successfully
```
## Viewing Connection Information
The `info` command displays connection details for the running environment.
```bash theme={null}
nile local info
```
### Examples
```bash Info Examples theme={null}
# Show connection information
nile local info
# Show with debug information
nile local info --debug
```
### Sample Output
```bash Info Output theme={null}
Connection Information:
Host: localhost
Port: 5432
Database: test
Username: 00000000-0000-0000-0000-000000000000
Password: password
```
## Common Workflows
### Starting Development Session
```bash Development Setup theme={null}
# 1. Start the environment
nile local start
# 2. Connect using psql (if prompted)
# Or use connection information with your preferred client
# 3. Start development
# 4. Stop when finished
nile local stop
```
### Using with Database Tools
```bash Database Tools theme={null}
# Get connection information
nile local info
# Use with your preferred database tool
# Example: TablePlus, DBeaver, pgAdmin, etc.
```
### Debugging Connection Issues
```bash Debug Example theme={null}
# Start with debug output
nile local start --debug
# Check connection information
nile local info --debug
```
## Common Issues
### Container Already Running
```bash Container Running theme={null}
# Start attempt with existing container
nile local start
Error: A Nile local environment is already running
# Solution: Stop existing container
nile local stop
```
### PostgreSQL Client Missing
```bash Missing PSQL theme={null}
# When psql is not installed
Error: Failed to launch psql
# Solution: Install PostgreSQL client tools
# macOS:
brew install postgresql
# Ubuntu:
sudo apt-get install postgresql-client
```
### Docker Not Running
```bash Docker Not Running theme={null}
# When Docker daemon is not running
Error: Docker failed to start
# Solution: Start Docker daemon
# macOS: Start Docker Desktop
# Linux: sudo systemctl start docker
```
## Best Practices
1. **Clean Environment**: Always stop the environment when not in use:
```bash theme={null}
nile local stop
```
2. **Connection Management**: Save connection information for reuse:
```bash theme={null}
nile local info > connection.txt
```
3. **Debug Mode**: Use debug flag when encountering issues:
```bash theme={null}
nile local start --debug
```
## Related Commands
* `nile db` - Manage cloud databases
* `nile tenants` - Manage tenants
* `nile users` - Manage users
* `nile config` - Configure CLI settings
## Environment Variables
The local environment uses these default values:
```bash Environment theme={null}
POSTGRES_USER=00000000-0000-0000-0000-000000000000
POSTGRES_PASSWORD=password
POSTGRES_DB=test
POSTGRES_PORT=5432
```
## Security Notes
1. The local environment is for development only
2. Default credentials should never be used in production
3. The environment is not encrypted by default
4. Container is accessible only from localhost
# Managing Tenants
Source: https://thenile.dev/docs/cli/managing_tenants
Nile provides built-in multi-tenancy support. This guide covers all tenant management commands and their usage.
```bash theme={null}
nile tenants [command] [options]
```
Available commands:
* `list` - View all tenants
* `create` - Create a new tenant
* `delete` - Remove a tenant
* `update` - Modify tenant details
## Prerequisites
Before managing tenants, ensure you have:
1. Selected a workspace
2. Selected a database
3. Authenticated with Nile
```bash Prerequisites theme={null}
# Set workspace and database
nile config --workspace my-workspace --db my-database
# Verify configuration
nile config
```
## Listing Tenants
The `list` command shows all tenants in your database.
```bash theme={null}
nile tenants list [options]
```
### Options
| Flag | Description | Default |
| ------------- | --------------------------------- | ----------------- |
| `--format` | Output format (human, json, csv) | human |
| `--workspace` | Workspace containing the database | Current workspace |
| `--db` | Database to list tenants from | Current database |
### Examples
```bash List Examples theme={null}
# Basic list
nile tenants list
# JSON format
nile tenants list --format json
# List from specific database
nile tenants list --db customer-db
```
## Creating Tenants
The `create` command adds a new tenant to your database.
```bash theme={null}
nile tenants create [options]
```
### Required Options
| Flag | Description |
| -------- | ------------------ |
| `--name` | Name of the tenant |
### Additional Options
| Flag | Description | Default |
| ------------- | --------------------------------- | ----------------- |
| `--id` | Custom ID for the tenant | Auto-generated |
| `--format` | Output format (human, json, csv) | human |
| `--workspace` | Workspace containing the database | Current workspace |
| `--db` | Database to create tenant in | Current database |
### Examples
```bash Create Examples theme={null}
# Create with auto-generated ID
nile tenants create --name "Acme Corp"
# Create with custom ID
nile tenants create --name "Globex Inc" --id globex-2024
# Create with JSON output
nile tenants create --name "Initech" --format json
```
## Updating Tenants
The `update` command modifies existing tenant details.
```bash theme={null}
nile tenants update [options]
```
### Required Options
| Flag | Description |
| ------------ | -------------------------- |
| `--id` | ID of the tenant to update |
| `--new_name` | New name for the tenant |
### Additional Options
| Flag | Description | Default |
| ------------- | --------------------------------- | ----------------- |
| `--format` | Output format (human, json, csv) | human |
| `--workspace` | Workspace containing the database | Current workspace |
| `--db` | Database containing the tenant | Current database |
### Examples
```bash Update Examples theme={null}
# Update tenant name
nile tenants update --id tenant_123 --new_name "Acme Corporation"
# Update with JSON output
nile tenants update --id tenant_123 --new_name "Acme Corp Ltd" --format json
# Update in specific database
nile tenants update --id tenant_123 --new_name "New Name" --db customer-db
```
## Deleting Tenants
The `delete` command removes a tenant from your database.
```bash theme={null}
nile tenants delete [options]
```
### Required Options
| Flag | Description |
| ------ | -------------------------- |
| `--id` | ID of the tenant to delete |
### Additional Options
| Flag | Description | Default |
| ------------- | --------------------------------- | ----------------- |
| `--force` | Skip confirmation prompt | false |
| `--workspace` | Workspace containing the database | Current workspace |
| `--db` | Database containing the tenant | Current database |
### Examples
```bash Delete Examples theme={null}
# Delete with confirmation
nile tenants delete --id tenant_123
# Force delete without confirmation
nile tenants delete --id tenant_123 --force
# Delete from specific database
nile tenants delete --id tenant_123 --db customer-db
```
## Common Workflows
### Setting Up Initial Tenants
```bash Initial Setup theme={null}
# 1. Create main tenant
nile tenants create --name "Main Customer"
# 2. Create additional tenants
nile tenants create --name "Customer A"
nile tenants create --name "Customer B"
# 3. Verify creation
nile tenants list
```
### Tenant Maintenance
```bash Maintenance theme={null}
# 1. List all tenants
nile tenants list
# 2. Update names if needed
nile tenants update --id tenant_123 --new_name "Updated Name"
# 3. Remove unused tenants
nile tenants delete --id tenant_456 --force
```
### Bulk Operations
```bash Bulk Operations [expandable] theme={null}
# List all tenants in JSON format
nile tenants list --format json > tenants.json
# Create multiple tenants
cat tenants.txt | while read name; do
nile tenants create --name "$name"
done
# Update multiple tenants
cat updates.txt | while read id name; do
nile tenants update --id "$id" --new_name "$name"
done
```
## Common Issues
### Tenant Not Found
When tenant doesn't exist:
```bash Not Found theme={null}
# Try to update non-existent tenant
nile tenants update --id invalid_id --new_name "New Name"
Error: Tenant 'invalid_id' not found
# List available tenants
nile tenants list
```
### Database Not Selected
When database isn't specified:
```bash No Database theme={null}
# Try without database selected
nile tenants list
Error: No database specified
# Solution: Set database
nile config --db my-database
```
### Duplicate Tenant ID
When creating tenant with existing ID:
```bash Duplicate ID theme={null}
# Try to create with existing ID
nile tenants create --name "New Tenant" --id existing_id
Error: Tenant ID 'existing_id' already exists
# Use auto-generated ID instead
nile tenants create --name "New Tenant"
```
## Best Practices
1. **Naming Conventions**: Use clear, consistent tenant names:
```bash theme={null}
nile tenants create --name "Company Name - Environment"
```
2. **Regular Verification**: Periodically verify tenant list:
```bash theme={null}
nile tenants list --format json > tenant_audit.json
```
3. **Backup Before Delete**: List tenant details before deletion:
```bash theme={null}
nile tenants list > tenants_backup.txt
```
## Related Commands
* `nile db psql` - Connect to tenant's database
* `nile db connectionstring` - Get tenant-specific connection
* `nile config` - Configure CLI settings
* `nile workspace` - Manage workspaces
# Managing Workspaces
Source: https://thenile.dev/docs/cli/managing_workspaces
Workspaces in Nile help you organize your databases and resources. This guide covers the commands available for managing workspaces.
```bash theme={null}
nile workspace [command] [options]
```
Available commands:
* `list` - View all available workspaces
* `show` - Display current workspace details
## Setting Active Workspace
Before using workspace commands, you can set your active workspace:
```bash theme={null}
nile config --workspace
```
You can also use the `NILE_WORKSPACE` environment variable:
```bash Environment Variable theme={null}
export NILE_WORKSPACE=production
```
## Listing Workspaces
The `list` command shows all workspaces you have access to.
```bash theme={null}
nile workspace list [options]
```
### Options
| Flag | Description | Default |
| ---------- | -------------------------------- | ------- |
| `--format` | Output format (human, json, csv) | human |
### Examples
```bash List Examples theme={null}
# Basic list
nile workspace list
# JSON format
nile workspace list --format json
# CSV format
nile workspace list --format csv
```
## Showing Current Workspace
The `show` command displays details about your current workspace.
```bash theme={null}
nile workspace show [options]
```
### Options
| Flag | Description | Default |
| ---------- | -------------------------------- | ------- |
| `--format` | Output format (human, json, csv) | human |
### Examples
```bash Show Examples theme={null}
# Show current workspace
nile workspace show
# JSON format
nile workspace show --format json
# CSV format
nile workspace show --format csv
```
### Sample Output
## Common Workflows
### Setting Up Development Environment
```bash Development Setup theme={null}
# 1. List available workspaces
nile workspace list
# 2. Set active workspace
nile config --workspace development
# 3. Verify workspace
nile workspace show
# 4. List databases in workspace
nile db list
```
### Workspace Verification
```bash Verification Workflow theme={null}
# 1. Check current workspace
nile workspace show
# 2. List all workspaces
nile workspace list
# 3. Verify access
nile db list # Lists databases in current workspace
```
## Common Issues
### No Active Workspace
When no workspace is selected:
```bash No Workspace theme={null}
# Try command without workspace
nile db list
Error: No workspace specified
# Solution: Set workspace
nile config --workspace development
```
### Authentication Issues
When authentication is required:
```bash Auth Issues theme={null}
# Authentication error
nile workspace list
Error: Authentication required
# Solution: Login first
nile connect login
```
### Invalid Workspace
When workspace doesn't exist:
```bash Invalid Workspace theme={null}
# Try invalid workspace
nile config --workspace nonexistent
Error: Workspace 'nonexistent' not found
# Solution: List available workspaces
nile workspace list
```
## Best Practices
1. **Environment Variables**: Use environment variables for different contexts:
```bash theme={null}
# Development
export NILE_WORKSPACE=development
# Production
export NILE_WORKSPACE=production
```
2. **Configuration File**: Set default workspace in configuration:
```bash theme={null}
nile config --workspace default-workspace
```
3. **Verification**: Always verify workspace before critical operations:
```bash theme={null}
nile workspace show
```
## Related Commands
* `nile config` - Configure CLI settings
* `nile db list` - List databases in workspace
* `nile connect` - Authentication management
* `nile tenants` - Tenant management within workspace
# Setting Configurations
Source: https://thenile.dev/docs/cli/setting_configs
The `nile config` command helps you manage your CLI configuration settings. This guide covers all configuration options and their usage.
```bash theme={null}
nile config [options]
```
The configuration system supports:
* Viewing current settings
* Setting individual values
* Setting multiple values at once
* Resetting to defaults
* Environment variable overrides
## Viewing Configuration
View your current configuration settings:
```bash theme={null}
nile config
```
## Setting API Key
Set your API key for authentication:
```bash theme={null}
nile config --api-key
```
### Examples
```bash API Key Examples theme={null}
# Set API key
nile config --api-key sk_test_1234...
# Set API key from environment variable
export NILE_API_KEY=sk_test_1234...
nile config
```
## Setting Workspace
Set your default workspace:
```bash theme={null}
nile config --workspace
```
### Examples
```bash Workspace Examples theme={null}
# Set default workspace
nile config --workspace development
# Set workspace and verify
nile config --workspace production
nile workspace show
```
## Setting Database
Set your default database:
```bash theme={null}
nile config --db
```
### Examples
```bash Database Examples theme={null}
# Set default database
nile config --db customer-db
# Set database and verify
nile config --db myapp-db
nile db show
```
## Setting Multiple Configurations
Set multiple configuration values at once:
```bash theme={null}
nile config --api-key --workspace --db
```
### Examples
```bash Multiple Config Examples theme={null}
# Set API key and workspace
nile config --api-key sk_test_1234... --workspace development
# Set workspace and database
nile config --workspace staging --db test-db
# Set all main configurations
nile config --api-key sk_test_1234... --workspace production --db prod-db
```
## Resetting Configuration
Reset all settings to their default values:
```bash theme={null}
nile config reset
```
### Examples
```bash Reset Examples theme={null}
# Reset all configurations
nile config reset
# Reset and verify
nile config reset
nile config
```
## Environment Variables
All configuration settings can be set using environment variables:
```bash Environment Variables theme={null}
# Authentication
export NILE_API_KEY=sk_test_1234...
# Workspace and Database
export NILE_WORKSPACE=development
export NILE_DB=myapp-db
# Host Configuration
export NILE_DB_HOST=custom.db.host
export NILE_GLOBAL_HOST=custom.global.host
```
## Configuration Precedence
Settings are applied in the following order (highest to lowest priority):
1. Command-line flags
2. Environment variables
3. Configuration file
4. Default values
```bash Precedence Example theme={null}
# Environment variable set
export NILE_WORKSPACE=production
# Command-line flag overrides environment
nile config --workspace development
# Result: workspace is set to "development"
```
## Common Issues
### Invalid API Key
When API key is invalid:
```bash Invalid Key theme={null}
# Set invalid API key
nile config --api-key invalid_key
Error: Invalid API key format
# Solution: Use valid API key
nile config --api-key sk_test_1234...
```
### Workspace Not Found
When workspace doesn't exist:
```bash Invalid Workspace theme={null}
# Set non-existent workspace
nile config --workspace nonexistent
Error: Workspace 'nonexistent' not found
# Solution: List available workspaces
nile workspace list
```
### Database Not Found
When database doesn't exist:
```bash Invalid Database theme={null}
# Set non-existent database
nile config --db nonexistent
Error: Database 'nonexistent' not found
# Solution: List available databases
nile db list
```
## Best Practices
1. **Environment-specific Configurations**:
```bash theme={null}
# Development
nile config --workspace dev --db dev-db
# Production
nile config --workspace prod --db prod-db
```
2. **Configuration Verification**:
```bash theme={null}
# Set and verify
nile config --workspace development
nile workspace show
```
3. **Security Best Practices**:
```bash theme={null}
# Store API key in environment
export NILE_API_KEY=sk_test_1234...
# Use configuration file for non-sensitive settings
nile config --workspace development --db dev-db
```
## Related Commands
* `nile workspace` - Manage workspaces
* `nile db` - Manage databases
* `nile connect` - Authentication management
* `nile tenants` - Tenant management
# Using Authentication
Source: https://thenile.dev/docs/cli/using_auth
The `nile auth` commands help you set up and manage authentication in your applications. These commands streamline the process of adding multi-tenant authentication to your projects and managing authentication-related configurations.
## Overview
```bash theme={null}
nile auth quickstart --nextjs # Add Multi-tenant Authentication in 2 minutes
nile auth env # Generate environment variables
nile auth env --output .env.local # Save environment variables to a file
```
## Quick Start Guide
The quickstart command helps you create a new application with Nile authentication pre-configured. Currently supports Next.js applications.
### Creating a Next.js Application with Authentication
```bash theme={null}
nile auth quickstart --nextjs
```
This command will:
1. Create a new Next.js application
2. Set up database credentials
3. Install required Nile packages
4. Configure API routes
5. Add authentication components
6. Start the development server
### What Gets Created
The quickstart command sets up the following structure:
```
nile-app/
├── .env.local # Environment variables
├── app/
│ ├── page.tsx # Main page with auth components
│ └── api/
│ └── [...nile]/ # Nile API routes
│ ├── nile.ts # Nile configuration
│ └── route.ts # API route handlers
└── package.json # Project dependencies
```
#### Authentication Components
The command sets up a basic authentication page with:
```tsx theme={null}
import {
SignOutButton,
SignUpForm,
SignedIn,
SignedOut,
UserInfo,
} from '@niledatabase/react';
import '@niledatabase/react/styles.css';
export default function SignUpPage() {
return (
);
}
```
## Environment Variables
The `env` command helps you generate and manage authentication-related environment variables.
### Generating Environment Variables
```bash theme={null}
# Display variables in terminal
nile auth env
# Save variables to a file
nile auth env --output .env.local
```
### Generated Variables
The command creates the following environment variables:
```env theme={null}
NILEDB_USER=your_database_user
NILEDB_PASSWORD=your_database_password
NILEDB_API_URL=https://us-west-2.api.thenile.dev/v2/databases/your_database_id
NILEDB_POSTGRES_URL=postgres://us-west-2.db.thenile.dev:5432/your_database
```
## Best Practices
1. **Environment Variables**
* Always use `--output` flag to save variables to `.env.local`
* Add `.env.local` to your `.gitignore`
* Never commit sensitive credentials to version control
2. **Authentication Components**
* Customize the authentication UI to match your application's design
* Add additional fields to the sign-up form as needed
* Implement proper error handling
3. **Security**
* Keep your database credentials secure
* Regularly rotate credentials using `nile auth env`
* Use environment variables for all sensitive information
## Troubleshooting
### Common Issues
1. **Quickstart Command Fails**
```bash theme={null}
# Ensure you're connected to Nile
nile connect login
# Verify workspace and database are set
nile config
```
2. **Environment Variables Not Working**
```bash theme={null}
# Regenerate environment variables
nile auth env --output .env.local
# Restart your development server
npm run dev
```
3. **Authentication Components Not Showing**
```bash theme={null}
# Ensure Nile packages are installed
npm install @niledatabase/server @niledatabase/react @niledatabase/client
# Check API routes are properly configured
ls app/api/[...nile]/
```
## Related Commands
* `nile connect login` - Connect to Nile using browser-based authentication
* `nile local start` - Start local development environment
* `nile config` - Configure workspace and database settings
## Examples
### Basic Authentication Setup
```bash theme={null}
# Create new Next.js app with authentication
nile auth quickstart --nextjs
# Generate environment variables
nile auth env --output .env.local
# Start the development server
cd nile-app
npm run dev
```
### Custom Authentication Flow
1. Create a new Next.js app manually
```bash theme={null}
npx create-next-app@latest my-app
cd my-app
```
2. Install Nile packages
```bash theme={null}
npm install @niledatabase/server @niledatabase/react @niledatabase/client
```
3. Generate environment variables
```bash theme={null}
nile auth env --output .env.local
```
4. Set up API routes and components manually
```bash theme={null}
mkdir -p app/api/[...nile]
# Add nile.ts and route.ts files
# Customize authentication components
```
## API Reference
### `nile auth quickstart`
Options:
* `--nextjs` - Create a Next.js application (required)
### `nile auth env`
Options:
* `--output ` - Save variables to a file (optional)
## Further Reading
* [Nile Authentication Documentation](https://thenile.dev/docs/authentication)
* [Next.js Integration Guide](https://thenile.dev/docs/integrations/nextjs)
* [Custom Authentication Flows](https://thenile.dev/docs/authentication/custom)
```
```
# Bloom
Source: https://thenile.dev/docs/extensions/bloom
Probabilistic index that can be useful for columns with many distinct values.
The [Bloom Index](https://www.postgresql.org/docs/current/bloom.html) in PostgreSQL is a type of index that can be useful for columns with many distinct values. It is particularly effective when searching across multiple indexed columns using equality queries.
Your Nile database arrives with `bloom` extension already enabled, so there's no need to run `create extension`.
## Creating and Populating Sample Table
Before creating the index, let's create a sample table and populate it with data:
```sql theme={null}
CREATE TABLE my_table (
id SERIAL PRIMARY KEY,
column1 TEXT,
column2 TEXT
);
INSERT INTO my_table (column1, column2) VALUES
('apple', 'red'),
('banana', 'yellow'),
('grape', 'purple'),
('orange', 'orange'),
('lemon', 'yellow');
```
## Creating a Bloom Index
A Bloom index is most useful for queries that filter on multiple columns. You need to specify the indexed columns and configure the number of bits per column (`colN`):
```sql theme={null}
CREATE INDEX bloom_idx ON my_table
USING bloom (column1, column2)
WITH (col1 = 4, col2 = 4);
```
Note: `USING bloom` specifies that this is a Bloom filter index and `col1 = 4,
col2 = 4` defines the number of bits per column to be used in the index
(default is 4).
## Querying with Bloom Index
Once the Bloom index is created, it can be used to optimize queries with equality conditions on indexed columns:
```sql theme={null}
SELECT * FROM my_table WHERE column1 = 'value1' AND column2 = 'value2';
```
## Limitations
* Bloom filters are probabilistic and can produce **false positives**, meaning they may return more results than expected.
* They are best suited for queries filtering multiple indexed columns using **equality conditions** (`=`).
* Unlike B-tree indexes, they **do not support range queries** (`<`, `>`, `BETWEEN`).
## Removing a Bloom Index
If you need to remove a Bloom index:
```sql theme={null}
DROP INDEX bloom_idx;
```
## Conclusion
Bloom indexes in PostgreSQL are useful for multi-column searches with high-cardinality data. They offer space efficiency but come with some trade-offs, such as potential false positives and lack of range query support.
For more details, refer to the [PostgreSQL documentation](https://www.postgresql.org/docs/current/bloom.html).
# Btree_gin
Source: https://thenile.dev/docs/extensions/btree_gin
Enables GIN indexes to support B-tree indexable data types.
The [btree\_gin](https://www.postgresql.org/docs/current/btree-gin.html) extension in PostgreSQL enables GIN indexes to support B-tree indexable data types. It is useful when you want to use a GIN index for multi-column queries that include standard B-tree searchable data types like `int`, `text`, `timestamp`, and `uuid`.
Your Nile database arrives with `btree_gin` extension already enabled, so there's no need to run `create extension`.
## Creating and Populating a Sample Table
Before creating the index, let's create a sample table and populate it with data:
```sql theme={null}
CREATE TABLE my_table (
id SERIAL PRIMARY KEY,
column1 TEXT,
column2 INT,
column3 TIMESTAMP
);
INSERT INTO my_table (column1, column2, column3) VALUES
('apple', 10, '2024-01-01 10:00:00'),
('banana', 20, '2024-02-01 11:30:00'),
('grape', 15, '2024-03-01 14:15:00'),
('orange', 25, '2024-04-01 09:45:00'),
('lemon', 30, '2024-05-01 16:20:00');
```
## Creating a GIN Index with btree\_gin
A GIN index with `btree_gin` is useful when performing multi-column searches that include B-tree indexable columns. Here's how to create one:
```sql theme={null}
CREATE INDEX gin_idx ON my_table
USING gin (column1, column2, column3);
```
Note: `USING gin` specifies that this is a GIN index. The extension allows
`column1` (text), `column2` (integer), and `column3` (timestamp) to be indexed
efficiently using GIN.
## Querying with GIN Index
Once the GIN index is created, it can be used to optimize queries filtering on indexed columns:
```sql theme={null}
SELECT * FROM my_table WHERE column1 = 'apple' AND column2 = 10;
```
## Limitations
* GIN indexes are optimized for **fast lookups** but have **slower insert/update performance** compared to B-tree indexes.
* They work best when querying multiple indexed columns together.
* Unlike B-tree indexes, they **do not support range queries efficiently**.
## Removing a GIN Index
If you need to remove a GIN index:
```sql theme={null}
DROP INDEX gin_idx;
```
## Conclusion
The `btree_gin` extension enhances GIN indexes by allowing them to handle B-tree indexable data types efficiently. It is particularly useful for **multi-column indexing** scenarios where a mix of text, integer, and timestamp fields are queried together.
For more details, refer to the [PostgreSQL documentation](https://www.postgresql.org/docs/current/btree-gin.html).
# Btree_gist
Source: https://thenile.dev/docs/extensions/btree_gist
Allows GiST indexes to support B-tree indexable data types.
The [btree\_gist](https://www.postgresql.org/docs/current/btree-gist.html) extension in PostgreSQL allows GiST indexes to support B-tree indexable data types. It is useful for indexing columns that typically use B-tree indexes but require additional GiST-specific features such as **multicolumn indexing**, **range queries**, and **support for exclusion constraints**.
Your Nile database arrives with `btree_gist` extension already enabled, so there's no need to run `create extension`.
## Creating and Populating a Sample Table
Before creating the index, let's create a sample table and populate it with data:
```sql theme={null}
CREATE TABLE my_table (
id SERIAL PRIMARY KEY,
column1 TEXT,
column2 INT,
column3 TSTZRANGE
);
INSERT INTO my_table (column1, column2, column3) VALUES
('apple', 10, '[2024-01-01 10:00:00, 2024-01-01 12:00:00]'),
('banana', 20, '[2024-02-01 11:30:00, 2024-02-01 13:30:00]'),
('grape', 15, '[2024-03-01 14:15:00, 2024-03-01 16:15:00]'),
('orange', 25, '[2024-04-01 09:45:00, 2024-04-01 11:45:00]'),
('lemon', 30, '[2024-05-01 16:20:00, 2024-05-01 18:20:00]');
```
## Creating a GiST Index with btree\_gist
A GiST index with `btree_gist` can be used for multi-column searches and exclusion constraints. Here's how to create one:
```sql theme={null}
CREATE INDEX gist_idx ON my_table
USING gist (column1, column2, column3);
```
Note: `USING gist` specifies that this is a GiST index. The extension allows
`column1` (text), `column2` (integer), and `column3` (timestamp) to be indexed
efficiently using GiST.
## Querying with GiST Index
Once the GiST index is created, it can be used to optimize queries filtering on indexed columns:
```sql theme={null}
SELECT * FROM my_table WHERE column1 = 'apple' AND column2 = 10;
```
## Limitations
* GiST indexes generally **do not provide the same performance as B-tree indexes** for single-column lookups.
* They **excel at multi-column queries** and range searches but can be **slower for simple equality lookups**.
* `btree_gist` is useful primarily for **exclusion constraints** rather than improving query performance.
## Removing a GiST Index
If you need to remove a GiST index:
```sql theme={null}
DROP INDEX gist_idx;
```
## Conclusion
The `btree_gist` extension enhances GiST indexes by allowing them to handle B-tree indexable data types efficiently. It is particularly useful for **multi-column indexing**, **range queries**, and **exclusion constraints**.
For more details, refer to the [PostgreSQL documentation](https://www.postgresql.org/docs/current/btree-gist.html).
# Citext
Source: https://thenile.dev/docs/extensions/citext
Provides a case-insensitive text type.
The [citext](https://www.postgresql.org/docs/current/citext.html) extension in PostgreSQL provides a case-insensitive text type. It behaves just like the standard `TEXT` data type but treats values as case-insensitive when comparing or indexing, making it useful for case-insensitive searches and unique constraints.
Your Nile database arrives with `citext` extension already enabled, so there's no need to run `create extension`.
## Creating and Populating a Sample Table
Before creating the index, let's create a sample table and populate it with data:
```sql theme={null}
CREATE TABLE my_table (
id SERIAL PRIMARY KEY,
column1 CITEXT,
column2 INT
);
INSERT INTO my_table (column1, column2) VALUES
('Apple', 10),
('banana', 20),
('GRAPE', 15),
('Orange', 25),
('Lemon', 30);
```
## Benefits of Using citext
* Allows case-insensitive text comparisons without using `LOWER()`.
* Simplifies case-insensitive unique constraints and indexes.
* Reduces errors when working with user-provided text data like emails or usernames.
## Creating an Index on citext Columns
A B-tree index can be created on a `CITEXT` column just like a `TEXT` column:
```sql theme={null}
CREATE INDEX citext_idx ON my_table (column1);
```
This index will allow efficient case-insensitive lookups.
## Enforcing Unique Constraints with citext
A `UNIQUE` constraint on a `CITEXT` column ensures case-insensitive uniqueness:
```sql theme={null}
ALTER TABLE my_table ADD CONSTRAINT unique_column1 UNIQUE (column1);
```
Now, inserting values like `'APPLE'` or `'apple'` would result in a constraint violation.
## Querying with citext
Once the `citext` extension is enabled, queries automatically become case-insensitive:
```sql theme={null}
SELECT * FROM my_table WHERE column1 = 'aPpLe';
```
This will return the same result as searching for `'Apple'`, `'APPLE'`, or `'apple'`.
## Limitations
* `CITEXT` is slightly **slower** than `TEXT` due to case normalization.
* It does **not support LIKE queries** efficiently unless you create a functional index using `LOWER(column1)`.
* Collation-sensitive operations may not always behave as expected.
## Removing a citext Index
If you need to remove an index:
```sql theme={null}
DROP INDEX citext_idx;
```
## Conclusion
The `citext` extension in PostgreSQL simplifies case-insensitive text handling, making it ideal for usernames, emails, and other text fields where case differences should be ignored.
For more details, refer to the [PostgreSQL documentation](https://www.postgresql.org/docs/current/citext.html).
# Cube
Source: https://thenile.dev/docs/extensions/cube
Provides a data type for multi-dimensional cubes.
The [cube](https://www.postgresql.org/docs/current/cube.html) extension in PostgreSQL provides a data type for multi-dimensional cubes. It is useful for applications requiring vector operations, such as geometric data, multi-dimensional indexing, and scientific computing.
Your Nile database arrives with `cube` extension already enabled, so there's no need to run `create extension`.
## Creating and Populating a Sample Table
Before creating the index, let's create a sample table and populate it with cube data:
```sql theme={null}
CREATE TABLE my_table (
id SERIAL PRIMARY KEY,
dimensions CUBE
);
INSERT INTO my_table (dimensions) VALUES
('(1,2,3)'),
('(4,5,6)'),
('(7,8,9)'),
('(10,11,12)'),
('(13,14,15)');
```
## Querying cube Data
With the `cube` extension, you can perform various operations on multi-dimensional data:
### Find Points within a Specific Range
```sql theme={null}
SELECT * FROM my_table WHERE dimensions <@ '(0,0,0),(5,5,5)';
```
This query returns all points within the cube defined by `(0,0,0)` to `(5,5,5)`.
### Compute the Distance Between Two Points
```sql theme={null}
SELECT cube_distance('(1,2,3)', '(4,5,6)');
```
This function computes the Euclidean distance between two cube points.
## Creating an Index on cube Columns
To optimize queries, you can create a GiST index:
```sql theme={null}
CREATE INDEX cube_idx ON my_table USING gist (dimensions);
```
This index improves performance for queries filtering cube data.
## Limitations
* The `cube` type supports up to 100 dimensions by default.
* It does not support operations like `+`, `-`, or `*` directly; you must use provided cube functions.
* Indexing performance depends on the number of dimensions and the dataset size.
## Removing a cube Index
If you need to remove an index:
```sql theme={null}
DROP INDEX cube_idx;
```
## Conclusion
The `cube` extension in PostgreSQL enables efficient storage and querying of multi-dimensional data. It is particularly useful for geometric and scientific applications where vector operations and spatial indexing are needed.
For more details, refer to the [PostgreSQL documentation](https://www.postgresql.org/docs/current/cube.html).
# Earthdistance
Source: https://thenile.dev/docs/extensions/earthdistance
Provides functions for calculating great-circle distances between points on Earth.
The [earthdistance](https://www.postgresql.org/docs/current/earthdistance.html) extension in PostgreSQL provides functions for calculating great-circle distances between points on Earth. It is useful for applications that require geospatial distance calculations, such as location-based services and mapping applications.
Your Nile database arrives with `earthdistance` extension and its dependency `cube` already enabled, so there's no need to run `create extension`.
## Creating and Populating `locations` Table
Before performing distance calculations, let's create a sample table to store latitude and longitude values:
```sql theme={null}
CREATE TABLE locations (
id SERIAL PRIMARY KEY,
name TEXT,
latitude DOUBLE PRECISION,
longitude DOUBLE PRECISION
);
INSERT INTO locations (name, latitude, longitude) VALUES
('San Francisco', 37.7749, -122.4194),
('New York', 40.7128, -74.0060),
('Los Angeles', 34.0522, -118.2437),
('Chicago', 41.8781, -87.6298),
('Miami', 25.7617, -80.1918);
```
## Calculating Distance Between Two Points
To compute the great-circle distance between two points (given in latitude and longitude in degrees), use the `earth_distance()` function:
```sql theme={null}
SELECT a.name AS location_a, b.name AS location_b,
earth_distance(ll_to_earth(a.latitude, a.longitude), ll_to_earth(b.latitude, b.longitude)) AS distance_meters
FROM locations a, locations b
WHERE a.name = 'San Francisco' AND b.name = 'New York';
```
This returns the approximate distance in meters between San Francisco and New York.
## Finding Locations Within a Given Radius
To find all locations within 1000 km of San Francisco:
```sql theme={null}
SELECT name, earth_distance(ll_to_earth(37.7749, -122.4194), ll_to_earth(latitude, longitude)) AS distance_meters
FROM locations
WHERE earth_distance(ll_to_earth(37.7749, -122.4194), ll_to_earth(latitude, longitude)) < 1000000;
```
## Limitations
* `earthdistance` assumes a **spherical Earth model**, which may introduce minor inaccuracies.
* Distance calculations are **approximate** and may not be suitable for high-precision geospatial applications.
* Requires both `cube` and `earthdistance` extensions to be installed.
## Removing an Index
If you need to remove the spatial index:
```sql theme={null}
DROP INDEX locations_earth_idx;
```
## Conclusion
The `earthdistance` extension in PostgreSQL simplifies great-circle distance calculations for geographic coordinates. It is useful for applications needing fast location-based searches and distance queries.
For more details, refer to the [PostgreSQL documentation](https://www.postgresql.org/docs/current/earthdistance.html).
# Emailaddr
Source: https://thenile.dev/docs/extensions/emailaddr
Provides a specialized data type for storing and validating email addresses.
The [emailaddr](https://github.com/petere/pgemailaddr) extension in PostgreSQL provides a specialized data type for storing and validating email addresses. This extension ensures proper email formatting and allows efficient querying.
Your Nile database arrives with `emailaddr` extension already installed and enabled.
## Creating and Populating `contacts` Table
Before using `emailaddr`, let's create a sample table to store email addresses of key contacts:
```sql theme={null}
CREATE TABLE contacts (
tenant_id UUID,
name TEXT,
email EMAILADDR,
PRIMARY KEY (tenant_id, email)
);
-- Create a tenant first
INSERT INTO tenants (id, name) VALUES
('11111111-1111-1111-1111-111111111111', 'Example Corp');
INSERT INTO contacts (tenant_id, name, email) VALUES
('11111111-1111-1111-1111-111111111111', 'Alice', 'alice@example.com'),
('11111111-1111-1111-1111-111111111111', 'Bob', 'bob@example.net'),
('11111111-1111-1111-1111-111111111111', 'Charlie', 'charlie@example.org');
```
## Querying Email Addresses
Since `EMAILADDR` is a specialized type, queries can be performed as with standard text columns:
```sql theme={null}
SELECT * FROM contacts WHERE email = 'alice@example.com';
```
## Validating Email Format
The `emailaddr` type ensures that only valid email addresses are stored. For example, the following insertion will fail:
```sql theme={null}
INSERT INTO contacts (tenant_id, name, email) VALUES ('11111111-1111-1111-1111-111111111111','Invalid User', 'not_an_email');
```
This will result in an error because `'not_an_email'` is not a properly formatted email address.
## Limitations
* `EMAILADDR` ensures correct email formatting but does **not** verify if the email exists.
* Cannot store internationalized email addresses using Unicode.
## Removing an Index
If you need to remove the index:
```sql theme={null}
DROP INDEX email_idx;
```
## Conclusion
The `emailaddr` extension in PostgreSQL simplifies email validation and storage while ensuring efficient indexing for lookup queries.
For more details, refer to the official PostgreSQL documentation or community resources.
# Financial
Source: https://thenile.dev/docs/extensions/financial
Provides specialized function for computing the Irregular Internal Rate of Return (XIRR).
The [financial](https://github.com/intgr/pg_financial) extension in PostgreSQL provides specialized functions for financial calculations, particularly for computing the **Irregular Internal Rate of Return (XIRR)**. It enables handling cash flows with irregular timing, similar to spreadsheet functions like **XIRR** in Excel or LibreOffice.
Your Nile database arrives with `financial` extension already enabled, so there's no need to run `create extension`.
## Creating and Populating `transactions` Table
Let's create a sample table to store financial transactions with dates and cash flows:
```sql theme={null}
CREATE TABLE transactions (
tenant_id UUID NOT NULL,
account TEXT,
amount NUMERIC,
transaction_date DATE
);
-- Create a tenant first, if it doesn't exist yet
INSERT INTO tenants (id, name) VALUES
('11111111-1111-1111-1111-111111111111', 'Example Corp');
INSERT INTO transactions (tenant_id, account, amount, transaction_date) VALUES
('11111111-1111-1111-1111-111111111111', 'Alice', -1000.00, '2024-01-01'),
('11111111-1111-1111-1111-111111111111', 'Alice', 200.00, '2024-02-01'),
('11111111-1111-1111-1111-111111111111', 'Alice', 300.00, '2024-03-01'),
('11111111-1111-1111-1111-111111111111', 'Alice', 500.00, '2024-06-01'),
('11111111-1111-1111-1111-111111111111', 'Alice', 700.00, '2024-12-01');
```
## Calculating the Irregular Internal Rate of Return (XIRR)
To compute the **XIRR** for an account using `pg_financial`:
```sql theme={null}
SELECT xirr(amount, transaction_date) AS irr
FROM transactions;
```
Note: `xirr(amount, transaction_date)` computes the **internal rate of
return** for cash flows occurring at irregular intervals. Negative amounts
typically represent **investments**, while positive amounts represent
**returns**.
## Providing an Initial Guess for XIRR
The guess argument is an optional initial guess. When omitted, the function will use annualized return as the guess,
which is usually reliable. This attempts to compute the **XIRR** starting with an initial guess of **10% (0.1)**.
```sql theme={null}
SELECT xirr(amount, transaction_date, 0.1) AS irr
FROM transactions;
```
## Conclusion
The `pg_financial` extension in PostgreSQL provides essential financial calculation capabilities, particularly for evaluating investment returns with irregular cash flows. It is useful for financial modeling and investment analytics.
For more details, refer to the [`pg_financial` GitHub repository](https://github.com/intgr/pg_financial).
# Fuzzystrmatch
Source: https://thenile.dev/docs/extensions/fuzzystrmatch
Provides functions for fuzzy string matching.
The [`fuzzystrmatch`](https://www.postgresql.org/docs/current/fuzzystrmatch.html) extension in PostgreSQL provides functions for fuzzy string matching. It is useful for approximate string comparisons, spell-checking, and searching similar words in a database.
Your Nile database arrives with `fuzzystrmatch` extension already enabled.
## Available Functions
The `fuzzystrmatch` extension provides several functions for different types of string matching algorithms:
### Soundex
The `soundex()` function returns a four-character Soundex code based on how a word sounds:
```sql theme={null}
SELECT soundex('example');
```
### Difference
The `difference()` function compares two Soundex codes and returns a similarity score from 0 to 4 (higher means more similar):
```sql theme={null}
SELECT difference('example', 'exampel');
```
### Levenshtein Distance
The `levenshtein()` function computes the edit distance (number of single-character edits required to transform one string into another):
```sql theme={null}
SELECT levenshtein('example', 'exampel');
```
### Levenshtein Distance with Custom Costs
You can specify different costs for insertions, deletions, and substitutions:
```sql theme={null}
SELECT levenshtein('example', 'exampel', 1, 2, 1);
```
### Metaphone
The `metaphone()` function returns a phonetic representation of a word, useful for English-language fuzzy searches:
```sql theme={null}
SELECT metaphone('example', 10);
```
## Example: Finding Similar Names
If you have a table with names and want to find names similar to a given input:
```sql theme={null}
CREATE TABLE contacts (
tenant_id UUID,
name TEXT,
email EMAILADDR,
PRIMARY KEY (tenant_id, email)
);
-- Create a tenant first
INSERT INTO tenants (id, name) VALUES
('11111111-1111-1111-1111-111111111111', 'Example Corp');
INSERT INTO contacts (tenant_id, name, email) VALUES
('11111111-1111-1111-1111-111111111111', 'John', 'john@example.com'),
('11111111-1111-1111-1111-111111111111', 'Jon', 'jon@example.com'),
('11111111-1111-1111-1111-111111111111', 'Johnny', 'johnny@example.com'),
('11111111-1111-1111-1111-111111111111', 'Jonathan', 'jonathan@example.com');
SELECT name FROM contacts WHERE difference(name, 'Jon') > 2;
```
## Use Cases
* Finding similar names in a customer database.
* Detecting typos in text input.
* Enhancing search functionality with approximate string matching.
## Limitations
* Soundex is optimized for English and may not work well for other languages.
* Levenshtein distance can be computationally expensive for large datasets.
* Phonetic matching may not always align perfectly with intended pronunciations.
## Conclusion
The `fuzzystrmatch` extension provides multiple methods for fuzzy string matching, making it a valuable tool for approximate searches and typo detection in PostgreSQL databases.
For more details, refer to the [PostgreSQL documentation](https://www.postgresql.org/docs/current/fuzzystrmatch.html).
# H3
Source: https://thenile.dev/docs/extensions/h3
H3 is a spatial indexing system developed by Uber.
The [h3-pg](https://github.com/zachasme/h3-pg) extension in PostgreSQL provides support for the **H3 spatial indexing system**, developed by Uber. H3 divides the world into hexagonal cells, where each cell has a unique identifier (called an H3 cell ID or H3 index). These hexagonal cells enable efficient geospatial operations and proximity searches.
Your Nile database arrives with `H3` extension already installed and enabled.
## H3 Functions and Usage
The `h3` extension provides several functions for working with H3 hexagonal cells.
### Converting Latitude/Longitude to H3 Cell ID
You can convert a latitude/longitude coordinate into an H3 cell ID at a given resolution (0-15). Higher resolution numbers create smaller hexagons:
```sql theme={null}
SELECT h3_lat_lng_to_cell(POINT('37.7749, -122.4194'), 9);
```
### Getting the Center Coordinates of an H3 Cell
To retrieve the latitude and longitude of the center of an H3 cell:
```sql theme={null}
SELECT h3_cell_to_lat_lng('8928308280fffff');
```
### Finding Neighboring H3 Cells
To get the immediate neighbors of an H3 cell:
```sql theme={null}
SELECT h3_grid_disk('8928308280fffff', 1);
```
### Finding the H3 Resolution of a Cell
To determine the resolution of a given H3 cell index:
```sql theme={null}
SELECT h3_get_resolution('8928308280fffff');
```
### Finding the Parent or Child Cells
To get the parent or child H3 cells of a given resolution:
```sql theme={null}
SELECT h3_cell_to_parent('8928308280fffff', 8);
SELECT h3_cell_to_children('8928308280fffff', 10);
```
## Example: Storing H3 Indices in a Table
A typical use case involves storing H3 cell indices (or cell IDs) for geospatial queries:
```sql theme={null}
CREATE TABLE locations (
id SERIAL PRIMARY KEY,
name TEXT,
lat DOUBLE PRECISION,
lon DOUBLE PRECISION,
h3_index H3INDEX
);
INSERT INTO locations (name, lat, lon, h3_index)
VALUES ('San Francisco', 37.7749, -122.4194, h3_lat_lng_to_cell(Point('37.7749, -122.4194'), 9));
```
## Querying Locations by Proximity
Find all locations within a given H3 ring distance:
```sql theme={null}
SELECT l.name
FROM locations l
JOIN LATERAL (
SELECT h3_grid_disk(h3_lat_lng_to_cell(Point('37.7749, -122.4194'), 9), 1) AS h3_idx
) disk
ON l.h3_index = disk.h3_idx;
```
## Use Cases
* **Geospatial clustering** for mapping applications.
* **Proximity searches** for efficient location-based queries.
* **Hierarchical geospatial analysis** for different zoom levels.
## Limitations
* H3 cells are approximations of geographic areas and may not perfectly align with political boundaries.
* Higher-resolution cells result in significantly more data points, increasing storage and computational requirements.
## Conclusion
The `h3` extension in PostgreSQL enables powerful geospatial indexing and proximity searches using the H3 hexagonal grid system. It is particularly useful for geospatial applications requiring efficient location queries.
For more details, refer to the [`h3-pg` GitHub repository](https://github.com/bytesandbrains/h3-pg).
# H3_postgis
Source: https://thenile.dev/docs/extensions/h3_postgis
PostgreSQL extends H3 functionality with PostGIS integration.
Your Nile database arrives with `h3_postgis` extension and its dependencies `h3` and `postgis` already installed and enabled.
## H3\_postgis Functions and Usage
The `h3_postgis` extension provides several functions that enable interoperability between H3 and PostGIS.
### Converting a Geometry to an H3 Index
To convert a **PostGIS geometry** (Point) into an H3 index at a given resolution:
```sql theme={null}
SELECT h3_lat_lng_to_cell(ST_SetSRID(ST_MakePoint(-122.4194, 37.7749), 4326), 9);
```
### Converting an H3 Index to a Geometry
To convert an H3 index into a **PostGIS polygon geometry**:
```sql theme={null}
SELECT h3_cell_to_geometry('8928308280fffff');
```
### Finding Neighboring H3 Cells as Geometries
To get the neighboring H3 cells as **PostGIS geometries**:
```sql theme={null}
SELECT h3_cell_to_geometry(neighbor)
FROM h3_grid_disk('8928308280fffff', 1) AS neighbor;
```
### Checking if a Point is Within an H3 Cell
To check if a given point lies within an H3 indexed cell:
```sql theme={null}
SELECT ST_Contains(h3_cell_to_geometry('8928308280fffff'),
ST_SetSRID(ST_MakePoint(-122.4194, 37.7749), 4326));
```
## Example: Storing H3 Indices and Geometries in a Table
A common use case is storing H3 indices alongside geospatial data:
```sql theme={null}
CREATE TABLE locations (
id SERIAL PRIMARY KEY,
name TEXT,
geom GEOMETRY(Point, 4326),
h3_index H3INDEX
);
INSERT INTO locations (name, geom, h3_index)
VALUES ('San Francisco', ST_SetSRID(ST_MakePoint(-122.4194, 37.7749), 4326),
h3_lat_lng_to_cell(ST_SetSRID(ST_MakePoint(-122.4194, 37.7749), 4326), 9));
```
## Querying Locations by Proximity
To find all locations within a given H3 ring distance:
```sql theme={null}
SELECT DISTINCT l.name
FROM locations l
CROSS JOIN LATERAL (
SELECT h3_grid_disk(h3_lat_lng_to_cell(l.geom, 9), 1) AS nearby_cell
) disk
WHERE l.h3_index = disk.nearby_cell;
```
## Use Cases
* **Efficient geospatial indexing** using hexagonal grids.
* **Spatial analysis** integrating H3 with PostGIS functions.
* **Fast proximity searches** based on geospatial cells rather than bounding boxes.
## Limitations
* H3 cells approximate real-world geography and do not align with political or administrative boundaries.
* Higher-resolution H3 cells increase data granularity but also computational costs.
## Conclusion
The `h3_postgis` extension in PostgreSQL enables seamless integration between **H3 spatial indexing** and **PostGIS geometry operations**, making it an excellent choice for advanced geospatial analysis and location-based applications.
For more details, refer to the [`h3-pg` GitHub repository](https://github.com/bytesandbrains/h3-pg).
# HLL
Source: https://thenile.dev/docs/extensions/hll
Probabilistic data structure that estimates the number of unique elements in a set.
The [hll](https://github.com/citusdata/postgresql-hll) extension in PostgreSQL provides a **HyperLogLog** data structure for **approximate distinct counting**. It is highly efficient for estimating the number of unique elements in large datasets while using minimal memory.
Your Nile database arrives with `hll` extension already enabled.
Let's create a sample analytics system that efficiently tracks unique user interactions.
We'll use a fact table to store raw events and leverage **HLL (HyperLogLog)** for efficient approximate distinct
counting in our aggregated statistics. This pattern is common in analytics systems where you need to track unique users across different time periods while maintaining reasonable storage and query performance.
## Creating and Populating `events` Table
Let's create a sample table to store unique user interactions, using **HLL** for approximate distinct counts:
```sql theme={null}
-- First, create a tenant
INSERT INTO tenants (id, name) VALUES ('123e4567-e89b-12d3-a456-426614174000', 'Acme Corp');
-- Create a fact table for raw events
CREATE TABLE event_facts (
tenant_id UUID NOT NULL,
event_timestamp TIMESTAMP,
user_id INT,
event_type TEXT
);
-- Create an aggregated table with HLL for efficient unique counting
CREATE TABLE daily_event_stats (
tenant_id UUID NOT NULL,
event_date DATE,
event_type TEXT,
unique_users_hll HLL, -- Stores the set of all unique users for this day and event type
PRIMARY KEY (tenant_id, event_date, event_type)
);
-- Insert some raw events
INSERT INTO event_facts (tenant_id, event_timestamp, user_id, event_type)
VALUES
('123e4567-e89b-12d3-a456-426614174000', '2024-03-01 10:00:00', 1, 'click'),
('123e4567-e89b-12d3-a456-426614174000', '2024-03-01 10:05:00', 2, 'click'),
('123e4567-e89b-12d3-a456-426614174000', '2024-03-01 10:10:00', 3, 'click'),
('123e4567-e89b-12d3-a456-426614174000', '2024-03-01 10:15:00', 1, 'click'), -- duplicate user
('123e4567-e89b-12d3-a456-426614174000', '2024-03-02 10:00:00', 2, 'click'),
('123e4567-e89b-12d3-a456-426614174000', '2024-03-02 10:05:00', 4, 'click'),
('123e4567-e89b-12d3-a456-426614174000', '2024-03-02 10:10:00', 5, 'click');
-- Aggregate the raw events into daily stats using HLL
SET nile.tenant_id = '123e4567-e89b-12d3-a456-426614174000';
INSERT INTO daily_event_stats
SELECT
tenant_id,
date_trunc('day', event_timestamp)::DATE as event_date,
event_type,
hll_add_agg(hll_hash_integer(user_id))
FROM event_facts
GROUP BY tenant_id,date_trunc('day', event_timestamp)::DATE, event_type;
```
### Understanding HLL Hashing
Before values can be added to an HLL data structure, they must first be hashed. The `hll` extension provides several hashing functions for different data types:
* `hll_hash_integer(value)` - for integer values
* `hll_hash_text(value)` - for text values
* `hll_hash_bytea(value)` - for binary data
* `hll_hash_any(value)` - for other data types
These hash functions convert the input values into consistent hash values that the HLL algorithm can process. Hashing ensures that:
1. Values are uniformly distributed
2. The same input always produces the same hash
3. Different inputs are likely to produce different hashes
## Approximate Counting with HLL - Estimating the Number of Unique Users
Here's how to estimate the number of unique users for each event type across all dates.
Note that you can't simply add up the HLL values to get the total distinct count, as this would double-count users.
Instead, you need to use the `hll_union_agg` function:
```sql theme={null}
SELECT event_type, hll_cardinality(hll_union_agg(unique_users_hll)) AS estimated_unique_users
FROM daily_event_stats
GROUP BY event_type;
```
## Use Cases
* **Efficient distinct counting** for large-scale analytics.
* **Web traffic analysis** (e.g., unique visitors per day).
* **Approximate user engagement tracking**.
* **Optimized analytics dashboards** that require fast estimations.
## Limitations
* HyperLogLog provides an **approximate** distinct count, not an exact value.
* The accuracy of estimates depends on the **HLL precision settings**.
* Cannot retrieve individual elements once they are added to an HLL aggregate.
## Conclusion
The `hll` extension in PostgreSQL is an efficient solution for large-scale distinct counting, offering fast and memory-efficient approximations. It is particularly useful for analytics and tracking unique values over time.
For more details, refer to the [`hll` GitHub repository](https://github.com/citusdata/postgresql-hll).
# Hstore
Source: https://thenile.dev/docs/extensions/hstore
Store key-value pairs within a single PostgreSQL value
The `hstore` extension provides a data type for storing sets of key-value pairs within a single PostgreSQL value. This can be particularly useful when dealing with semi-structured data or when you need to store attributes that don't warrant their own columns.
Your Nile database arrives with `hstore` extension already enabled.
## Creating Tables with hstore
Here's how to create a table that includes an hstore column:
```sql theme={null}
CREATE TABLE products (
id int,
tenant_id uuid NOT NULL,
name text,
attributes hstore,
PRIMARY KEY(tenant_id, id)
);
```
### Inserting Data
First, create a tenant:
```sql theme={null}
INSERT INTO tenants (id, name)
VALUES ('d1c06023-3421-4fbb-9dd1-c96e42d2fd02', 'Example Store');
```
Then you can insert products for this tenant:
```sql theme={null}
-- Using the => operator
INSERT INTO products (tenant_id, id, name, attributes)
VALUES ('d1c06023-3421-4fbb-9dd1-c96e42d2fd02', 1, 'Laptop', 'color=>silver, RAM=>16GB, storage=>512GB');
-- Using the hstore function
INSERT INTO products (tenant_id, id, name, attributes)
VALUES ('d1c06023-3421-4fbb-9dd1-c96e42d2fd02', 2, 'Phone', hstore(ARRAY['color', 'black', 'storage', '256GB']));
```
### Querying hstore Data
The hstore data type comes with several useful operators and functions:
```sql theme={null}
-- Get a specific value by key
SELECT attributes->'color' as color FROM products;
-- Check if a key exists
SELECT * FROM products WHERE attributes ? 'RAM';
-- Check if a key/value pair exists
SELECT * FROM products WHERE attributes @> 'color=>silver';
-- Get all keys
SELECT akeys(attributes) FROM products;
-- Get all values
SELECT avals(attributes) FROM products;
-- Get key/value pairs as a set
SELECT skeys(attributes), svals(attributes) FROM products;
```
## Updating hstore Values
You can update individual key/value pairs or the entire hstore.
We use `SET nile.tenant_id` to guarantee the operation is performed on the correct tenant:
```sql theme={null}
SET nile.tenant_id = 'd1c06023-3421-4fbb-9dd1-c96e42d2fd02';
-- Update a single key/value pair
UPDATE products
SET attributes = attributes || 'RAM=>32GB'::hstore
WHERE id = 1;
-- Delete a key
UPDATE products
SET attributes = delete(attributes, 'storage')
WHERE id = 1;
```
## Combining hstore Values
You can combine multiple hstore values using the concatenation operator (||):
```sql theme={null}
SELECT 'a=>1, b=>2'::hstore || 'c=>3'::hstore;
```
## Converting To/From JSON
hstore can be converted to and from JSON:
```sql theme={null}
-- Convert hstore to JSON
SELECT hstore_to_json(attributes) FROM products;
-- Convert JSON to hstore using the array syntax
SELECT hstore(ARRAY['color', 'red', 'size', 'large']);
```
## Best Practices
1. Use hstore when dealing with dynamic attributes that don't require strict schema validation.
2. Consider using JSON/JSONB instead if you need to store nested structures or arrays.
3. Create indexes on frequently queried keys using GiST or GIN indexes:
```sql theme={null}
CREATE INDEX idx_products_attributes ON products USING GIN (attributes);
```
## Performance Considerations
* hstore is generally more efficient than JSON for simple key-value pairs
* GIN indexes can significantly improve query performance on hstore columns
* The storage size of hstore is typically smaller than equivalent JSON storage
## Limitations
* Keys and values must be text strings
* No support for nested structures
* No array support within values
* Maximum size is limited by the maximum TOAST size in PostgreSQL
For more details, refer to the [hstore documentation](https://www.postgresql.org/docs/current/hstore.html).
# Intagg
Source: https://thenile.dev/docs/extensions/intagg
Integer aggregator and enumerator functions
The `intagg` extension provides integer aggregation and enumeration functions, which are useful for working with integer arrays and creating sequences. This extension is particularly helpful when you need to aggregate integer values or generate enumerated lists.
Your Nile database arrives with the `intagg` extension already enabled.
## Available Functions
The extension provides two main functions:
1. `int_array_aggregate(integer)` - Aggregates integers into an array
2. `int_array_enum(integer[])` - Creates a set of integers from an array
## Integer Array Aggregation
Here's how to use the integer aggregation function:
```sql theme={null}
-- Create a table to store student scores
CREATE TABLE scores (
tenant_id uuid NOT NULL,
student_id integer,
subject text,
score integer,
PRIMARY KEY(tenant_id, student_id, subject)
);
INSERT INTO scores (tenant_id, student_id, subject, score) VALUES
('d1c06023-3421-4fbb-9dd1-c96e42d2fd02', 1, 'Math', 95),
('d1c06023-3421-4fbb-9dd1-c96e42d2fd02', 1, 'Science', 88),
('d1c06023-3421-4fbb-9dd1-c96e42d2fd02', 2, 'Math', 78),
('d1c06023-3421-4fbb-9dd1-c96e42d2fd02', 2, 'Science', 92);
-- Aggregate values into an array
SELECT int_array_aggregate(score) FROM scores;
-- Result: {95,88,78,92}
-- Get array of scores per student
SELECT student_id, int_array_aggregate(score) as scores
FROM scores
GROUP BY student_id;
```
## Integer Array Enumeration
The `int_array_enum` function unnesting aggregated arrays into rows. It is the opposite of `int_array_aggregate`.
```sql theme={null}
-- Convert array to rows
SELECT * FROM int_array_enum(ARRAY[1,2,3,4,5]);
-- Practical example: unnest aggregated scores
-- Pretend that scores are stored as an array of results per student, and we want to unnest them into rows
WITH aggregated_scores AS (
SELECT student_id, int_array_aggregate(score) as scores
FROM scores
GROUP BY student_id
)
SELECT student_id, unnested_score
FROM aggregated_scores,
int_array_enum(scores) as unnested_score;
```
## Performance Considerations
* The aggregation function is efficient for moderate-sized datasets
* For very large datasets, consider using native PostgreSQL array\_agg() function
* The enumeration function is useful for small to medium-sized arrays
## Limitations
* Works only with integer values
* Not suitable for very large arrays due to memory constraints
* The enumerated output maintains the order of the input array
## Alternative Approaches
For some use cases, you might want to consider these PostgreSQL native alternatives:
1. `array_agg()` for general-purpose array aggregation
2. `generate_series()` for generating sequences
3. `unnest()` for array expansion
For more details, refer to the [PostgreSQL documentation on aggregate functions](https://www.postgresql.org/docs/current/functions-aggregate.html).
The official documentation for the `intagg` extension is [here](https://www.postgresql.org/docs/current/intagg.html) and includes a nice example of
using the `intagg` functions to denormalize data and still be able to query it in a normalized way.
# Intarray
Source: https://thenile.dev/docs/extensions/intarray
Additional functions and operators for integer arrays
The `intarray` extension provides additional functions and operators for working with arrays of integers. It's particularly useful when you need to perform operations like unions, intersections, or searches on integer arrays without writing complex SQL queries.
Your Nile database arrives with the `intarray` extension already enabled.
## Operators
The extension provides several operators for array manipulation:
* `&&` - overlap (have elements in common)
* `@>` - contains
* `<@` - is contained by
* `=` - equal
* `+` - union
* `&` - intersection
* `-` - difference
## Basic Array Operations
Here's how to use the basic array operations:
```sql theme={null}
CREATE TABLE product_categories (
tenant_id uuid NOT NULL,
product_id integer,
category_ids integer[],
PRIMARY KEY(tenant_id, product_id)
);
-- Insert sample data
INSERT INTO product_categories (tenant_id, product_id, category_ids) VALUES
('d1c06023-3421-4fbb-9dd1-c96e42d2fd02', 1, '{1,2,3}'),
('d1c06023-3421-4fbb-9dd1-c96e42d2fd02', 2, '{2,4}'),
('d1c06023-3421-4fbb-9dd1-c96e42d2fd02', 3, '{1,3,4}');
-- Find products that have categories in common with product_id 1
SELECT product_id, category_ids
FROM product_categories
WHERE category_ids && (
SELECT category_ids
FROM product_categories
WHERE product_id = 1
)
AND product_id != 1;
-- Find products that contain all categories of product_id 2
SELECT product_id, category_ids
FROM product_categories
WHERE category_ids @> (
SELECT category_ids
FROM product_categories
WHERE product_id = 2
);
```
## Array Set Operations
The extension provides set operations for combining arrays:
```sql theme={null}
-- Union of two category sets
SELECT
'{1,2,3}'::int[] + '{3,4,5}'::int[] as union_result,
'{1,2,3}'::int[] & '{3,4,5}'::int[] as intersection_result,
'{1,2,3,4}'::int[] - '{3,4}'::int[] as difference_result;
-- Practical example: Find common categories between products
SELECT
p1.product_id as product1,
p2.product_id as product2,
p1.category_ids & p2.category_ids as common_categories
FROM product_categories p1
CROSS JOIN product_categories p2
WHERE p1.product_id < p2.product_id;
```
## Array Manipulation Functions
The extension includes several useful functions:
```sql theme={null}
-- Sort and remove duplicates
SELECT uniq(sort(category_ids))
FROM product_categories
WHERE product_id = 1;
-- Find the index of an element (1-based)
SELECT idx(category_ids, 3)
FROM product_categories
WHERE product_id = 1;
-- Add/remove elements
SELECT
category_ids || 5 as added_element, -- Standard array concatenation
category_ids || '{5,6}'::int[] as added_array
FROM product_categories
WHERE product_id = 1;
```
## Query Optimization
The extension supports GiST and GIN indexes for efficient array operations:
```sql theme={null}
-- Create GiST index
CREATE INDEX idx_category_gist ON product_categories USING gist (category_ids gist__int_ops);
-- Create GIN index (usually better for exact searches)
CREATE INDEX idx_category_gin ON product_categories USING gin (category_ids gin__int_ops);
```
These indexes can significantly improve performance for the following types of queries:
* Overlap (`&&`)
* Contains (`@>`)
* Contained by (`<@`)
* Equal (`=`)
## Performance Considerations
* GIN indexes are typically better for exact matches and contained-by queries
* GiST indexes are better for overlap queries but may be less precise
* Array operations are performed in memory, so be cautious with very large arrays
* Sorting and uniqueness operations (`sort`, `uniq`) create new arrays
## Limitations
* Works only with integer arrays
* No support for multi-dimensional arrays
* Array size is limited by available memory
* Some operations create copies of arrays, which can impact memory usage
## Alternative Approaches
For some use cases, you might want to consider:
1. Using junction tables for many-to-many relationships
2. Using native PostgreSQL array functions
3. Using JSONB arrays for more flexible data types
For more details, refer to the [PostgreSQL documentation on arrays](https://www.postgresql.org/docs/current/arrays.html) and the [intarray extension documentation](https://www.postgresql.org/docs/current/intarray.html).
# Introduction
Source: https://thenile.dev/docs/extensions/introduction
PostgreSQL Extension Store in Nile
PostgreSQL extensions are powerful add-ons that enhance PostgreSQL's functionality beyond its core features. They introduce new data types, functions, and operators to the database - usually to support a specific use case.
This rich ecosystem of extensions is available to you out of the box in Nile. No need to install or even run `CREATE EXTENSION` commands. Just use the extension in your queries.
You can experiment with the extensions and try them out in the [Nile console](https://console.thenile.dev), by navigating to
the `Extensions` tab on the left.
Don't see an extension you need? Let us know by creating a GitHub issue.
## Featured Extensions
Perform vector similarity search and build AI-powered features
Add support for geographic objects, allowing you to run location queries in
SQL
Encryption functions and hashing algorithms
UUID generation functions
## Search and Indexing
Probabilistic index that can be useful for columns with many distinct values
GIN index support for B-tree indexable data types.
GiST index support for B-tree indexable data types.
Case-insensitive text type
Functions for fuzzy string matching
DiskANN index support for pgvector
Full-text search using bigrams
Text similarity measures in PostgreSQL
Similarity search in PostgreSQL using trigrams
Prefix search functionality
## Statistical and Analytical
Data type for multi-dimensional cubes
Specialized function for computing the Irregular Internal Rate of Return
(XIRR)
HyperLogLog++ algorithm for estimating the number of unique elements in a
set
Integer aggregator and enumerator functions
Functions and operators for integer arrays
Efficient quantile and percentile calculations
Random data generator extension with wide range of data types
Cross tabulation and pivot operations
Incremental correlation calculations in PostgreSQL
## Geospatial
Functions for calculating great-circle distances between points on Earth.
Spatial indexing system developed by Uber
Extends H3 functionality with PostGIS integration
Geospatial routing extension for PostgreSQL
Raster Data Support for PostGIS
## Data Types and Storage
specialized data type for storing and validating email addresses
Store key-value pairs within a single PostgreSQL value
IPv4 and IPv6 types and range operations
International Standard Number data types
Hierarchical tree-like structures in PostgreSQL databases
Chemical structure types and functions
Line segment and floating-point interval data type
SI unit conversion and dimensional analysis
# IP4R
Source: https://thenile.dev/docs/extensions/ip4r
IPv4 and IPv6 types and range operations
The `ip4r` extension provides data types and functions for working with IPv4 and IPv6 addresses and ranges. It's particularly useful for network-related applications, IP-based access control, and geolocation services.
Your Nile database arrives with the `ip4r` extension already enabled.
## Data Types
The extension provides several data types:
* `ip4` - IPv4 address
* `ip4r` - IPv4 range
* `ip6` - IPv6 address
* `ip6r` - IPv6 range
* `ipaddress` - Can store either IPv4 or IPv6 address
* `iprange` - Can store either IPv4 or IPv6 range
## Basic Usage
Here's how to use the IP address types and operations:
```sql theme={null}
CREATE TABLE ip_access_rules (
tenant_id uuid NOT NULL,
rule_id integer NOT NULL,
network iprange,
description text,
is_allowed boolean,
PRIMARY KEY(tenant_id, rule_id)
);
-- Insert sample IPv4 rules
INSERT INTO ip_access_rules (tenant_id, rule_id, network, description, is_allowed) VALUES
('d1c06023-3421-4fbb-9dd1-c96e42d2fd02', 1, '10.0.0.0/8', 'Internal network', true),
('d1c06023-3421-4fbb-9dd1-c96e42d2fd02', 2, '192.168.1.0/24', 'Office network', true),
('d1c06023-3421-4fbb-9dd1-c96e42d2fd02', 3, '203.0.113.0/24', 'Blocked range', false);
-- Insert IPv6 rules
INSERT INTO ip_access_rules (tenant_id, rule_id, network, description, is_allowed) VALUES
('d1c06023-3421-4fbb-9dd1-c96e42d2fd02', 5, '2001:db8::/32', 'IPv6 documentation range', true),
('d1c06023-3421-4fbb-9dd1-c96e42d2fd02', 6, '2001:db8:1::/48', 'IPv6 subnet', false);
```
## IP Address Operations
The extension provides various operators for IP address manipulation and comparison:
```sql theme={null}
-- Check if an IP is in a range
SELECT network, description
FROM ip_access_rules
WHERE network >> '10.0.1.5'::ip4;
-- Find overlapping networks
SELECT a.network, a.description, b.network, b.description
FROM ip_access_rules a
JOIN ip_access_rules b ON a.network && b.network
WHERE a.rule_id < b.rule_id;
-- Get the containing network
SELECT network, description
FROM ip_access_rules
WHERE network >>= '192.168.1.0/24'::ip4r;
```
## Common Use Cases
### IP-based Access Control
```sql theme={null}
-- Check if an IP address is allowed (direct query)
SELECT EXISTS (
SELECT 1
FROM ip_access_rules
WHERE network >> '10.0.1.5'::ip4
AND is_allowed = true
);
-- Get all matching rules for an IP address
SELECT network, description, is_allowed
FROM ip_access_rules
WHERE network >> '10.0.1.5'::ip4
ORDER BY masklen(network) DESC -- Most specific match first
LIMIT 1;
-- Check multiple IPs at once
SELECT
client_ip,
EXISTS (
SELECT 1
FROM ip_access_rules
WHERE network >> client_ip::ipaddress
AND is_allowed = true
) as is_allowed
FROM (
VALUES
('10.0.1.5'),
('192.168.1.100'),
('203.0.113.1')
) as client_ips(client_ip);
```
### Network Range Analysis
```sql theme={null}
-- Find all subnets within a larger network
SELECT network, description
FROM ip_access_rules
WHERE network <<= '10.0.0.0/8'::ip4r;
-- Calculate number of addresses in each range
SELECT
network,
description,
CASE
WHEN family(network) = 4 THEN masklen(network::ip4r)
ELSE masklen(network::ip6r)
END as prefix_length
FROM ip_access_rules;
```
## Query Optimization
The extension supports GiST indexes for efficient range queries:
```sql theme={null}
-- Create GiST index
CREATE INDEX idx_ip_ranges ON ip_access_rules USING gist (network);
```
This index improves performance for these operators:
* `>>` (contains)
* `>>=` (contains or equals)
* `<<` (contained by)
* `<<=` (contained by or equals)
* `&&` (overlaps)
## Performance Considerations
* GiST indexes significantly improve range query performance
* IP address operations are very efficient as they use native integer comparisons
* Range operations are optimized for both IPv4 and IPv6
* Indexes work well with both IP versions in the same column
## Limitations
* Cannot mix IPv4 and IPv6 in range comparisons
* Some operations are version-specific (ip4 vs ip6)
* Maximum IPv4 range is /0 (0.0.0.0 to 255.255.255.255)
* Maximum IPv6 range is /0 (:: to ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff)
## Alternative Approaches
For some use cases, you might want to consider:
1. Using the built-in `inet` type for basic IP address storage
2. Using `cidr` type for network ranges without host bits
3. Using separate columns for IPv4 and IPv6 if operations are always version-specific
For more details, refer to the [PostgreSQL documentation on network address types](https://www.postgresql.org/docs/current/datatype-net-types.html) and
the [ip4r extension documentation](https://github.com/RhodiumToad/ip4r).
# ISN
Source: https://thenile.dev/docs/extensions/isn
International Standard Number data types
The `isn` extension provides data types for international product and publication numbering standards, including ISBN (books), ISMN (music), ISSN (serials), EAN13 (products), and UPC (products). It handles validation, formatting, and conversion between different standards.
Your Nile database arrives with the `isn` extension already enabled.
## Data Types
The extension provides several data types:
* `isbn` - International Standard Book Number (ISBN-13 and ISBN-10)
* `ismn` - International Standard Music Number
* `issn` - International Standard Serial Number
* `ean13` - European Article Number (includes UPC)
* `upc` - Universal Product Code
## Basic Usage
Here's how to use the ISN types with a product catalog:
```sql theme={null}
CREATE TABLE products (
tenant_id uuid NOT NULL,
id integer NOT NULL,
title text,
isbn isbn, -- For books
ean13 ean13, -- For general products
upc upc, -- For North American products
PRIMARY KEY(tenant_id, id)
);
CREATE TABLE publications (
tenant_id uuid NOT NULL,
id integer NOT NULL,
title text,
issn issn, -- For magazines/journals
ismn ismn, -- For music publications
PRIMARY KEY(tenant_id, id)
);
-- Insert sample book data
INSERT INTO products (tenant_id, id, title, isbn) VALUES
('d1c06023-3421-4fbb-9dd1-c96e42d2fd02', 1, 'Sample Book 1', '978-0-7475-3269-9'), -- ISBN-13
('d1c06023-3421-4fbb-9dd1-c96e42d2fd02', 2, 'Sample Book 2', '0-7475-3269-9'); -- ISBN-10
-- Insert sample product data
INSERT INTO products (tenant_id, id, title, ean13) VALUES
('d1c06023-3421-4fbb-9dd1-c96e42d2fd02', 3, 'Sample Product 1', '4006381333931'),
('d1c06023-3421-4fbb-9dd1-c96e42d2fd02', 4, 'Sample Product 2', '0012345678905');
-- Insert sample publication data
INSERT INTO publications (tenant_id, id, title, issn, ismn) VALUES
('d1c06023-3421-4fbb-9dd1-c96e42d2fd02', 1, 'Science Journal', '0317-8471', null),
('d1c06023-3421-4fbb-9dd1-c96e42d2fd02', 2, 'Music Score', null, 'M-2306-7118-7');
```
## Number Validation and Formatting
The ISN types automatically validate check digits and can handle various input formats:
```sql theme={null}
-- ISBN validation and formatting
SELECT
'978-0-7475-3269-9'::isbn as valid_isbn, -- Works
'978-0-7475-3269-X'::isbn as invalid_isbn; -- Fails check digit
-- Format conversion (ISBN-10 to ISBN-13)
SELECT
isbn13('0-7475-3269-9'::isbn) as isbn13,
isbn('978-0-7475-3269-9'::isbn) as isbn10;
-- EAN13/UPC validation
SELECT
'4006381333931'::ean13 as valid_ean, -- Works
'4006381333932'::ean13 as invalid_ean; -- Fails check digit
```
## Common Use Cases
### Product Lookup
```sql theme={null}
-- Look up a book by ISBN
SELECT title, isbn
FROM products
WHERE isbn = '978-0-7475-3269-9'::isbn;
```
### ISBN Range Management
```sql theme={null}
-- Find books in a specific ISBN publisher range
SELECT title, isbn
FROM products
WHERE
isbn IS NOT NULL
AND isbn13(isbn)::text LIKE '978-0-7475-%';
-- Group books by publisher prefix
SELECT
substring(isbn13(isbn)::text, 1, 8) as publisher_prefix,
count(*) as book_count
FROM products
WHERE isbn IS NOT NULL
GROUP BY publisher_prefix;
```
## Performance Considerations
* ISN types are stored efficiently as 64-bit integers internally
* Validation and check digit calculation is performed on input
* Indexes work efficiently with all ISN types
* Conversion between formats (e.g., ISBN-10/13) is fast
## Best Practices
1. Always use the appropriate type for each standard
2. Handle input format variations in your application
3. Use the built-in conversion functions rather than implementing your own
4. Consider indexing frequently searched ISN columns
For more details, refer to the [PostgreSQL ISN documentation](https://www.postgresql.org/docs/current/isn.html) and the relevant standards:
* [ISBN](https://www.isbn-international.org/)
* [ISSN](https://www.issn.org/)
* [ISMN](https://www.ismn-international.org/)
* [EAN](https://www.gs1.org/standards/barcodes/ean-upc)
# ltree
Source: https://thenile.dev/docs/extensions/ltree
Hierarchical tree-like structures in PostgreSQL databases
The `ltree` extension provides support for hierarchical tree-like structures in PostgreSQL databases. It's particularly useful for storing and querying hierarchical data such as organizational structures, file systems, or category trees.
Your Nile database arrives with the `ltree` extension already enabled.
## Understanding ltree
The ltree data type represents a label path - a sequence of labels separated by dots, like 'Top.Countries.USA.California'. Each label can include alphanumeric characters and underscores, with a maximum length of 256 bytes.
### Key Features
* **Hierarchical Data Storage**: Store tree-structured data in a single column
* **Efficient Querying**: Fast traversal and searching of tree structures
* **Path Manipulation**: Built-in operators for working with paths
* **Pattern Matching**: Powerful pattern matching capabilities
## Usage Examples
### Creating a Table with ltree
```sql theme={null}
CREATE TABLE categories (
id serial PRIMARY KEY,
path ltree
);
```
### Inserting Data
```sql theme={null}
INSERT INTO categories (path) VALUES
('Electronics'),
('Electronics.Computers'),
('Electronics.Computers.Laptops'),
('Electronics.Computers.Desktops'),
('Electronics.Phones'),
('Electronics.Phones.Smartphones');
```
### Querying Examples
Find all subcategories under 'Electronics':
```sql theme={null}
SELECT path FROM categories WHERE path <@ 'Electronics';
```
Find immediate children of 'Electronics.Computers':
```sql theme={null}
SELECT path FROM categories WHERE path ~ 'Electronics.Computers.*{1}';
```
Find the parent category:
```sql theme={null}
SELECT subpath(path, 0, -1) FROM categories WHERE path = 'Electronics.Computers.Laptops';
```
## Operators and Functions
### Common Operators
* `<@`: Is left argument a descendant of right (or equal)?
* `@>`: Is left argument an ancestor of right (or equal)?
* `~`: Does ltree match lquery?
* `?`: Does ltree match ltxtquery?
* `||`: Concatenate ltree paths
### Useful Functions
* `subpath(ltree, offset, len)`: Get subpath of ltree
* `nlevel(ltree)`: Return number of labels in path
* `index(ltree, ltree)`: Return position of second ltree in first
* `text2ltree(text)`: Cast text to ltree
* `ltree2text(ltree)`: Cast ltree to text
## Best Practices
1. **Plan Your Hierarchy**: Design your tree structure carefully before implementation
2. **Index Usage**: Create GiST indexes for better query performance:
```sql theme={null}
CREATE INDEX path_idx ON categories USING GIST (path);
```
3. **Validation**: Implement checks to maintain data integrity
4. **Path Length**: Keep paths reasonably short for better performance
## Use Cases
* Organization charts
* Product categories
* File system structures
* Location hierarchies
* Menu structures
* Content taxonomies
## Performance Considerations
* Use appropriate indexes based on your query patterns
* Monitor path lengths as very deep hierarchies can impact performance
* Consider denormalization for frequently accessed ancestor/descendant information
## Additional Resources
* [PostgreSQL ltree Documentation](https://www.postgresql.org/docs/current/ltree.html)
* [PostgreSQL Wiki - ltree](https://wiki.postgresql.org/wiki/Ltree)
# pg_bigm
Source: https://thenile.dev/docs/extensions/pg_bigm
Full-text search using bigrams
The `pg_bigm` extension provides fast full-text search functionality using 2-gram (bigram) matching in PostgreSQL databases. It's particularly useful when you need to perform similarity searches or fuzzy string matching on large text data.
Your Nile database arrives with the `pg_bigm` extension already enabled.
## Understanding pg\_bigm
A bigram is a pair of consecutive characters in a string. For example, the word "hello" contains the following bigrams: "he", "el", "ll", "lo". pg\_bigm creates an index of these bigrams, enabling fast similarity searches and partial matching queries.
### Key Features
* **Fast Full-Text Search**: Efficient searching using bigram matching
* **Similarity Calculation**: Built-in functions to measure string similarity
* **Partial Matching**: Find strings containing specific patterns
* **Language Agnostic**: Works well with any language, including non-Latin scripts
* **GIN Index Support**: Fast search performance using GIN indexes
## Usage Examples
### Creating a Table with Text Search
```sql theme={null}
CREATE TABLE articles (
tenant_id uuid,
id integer,
title text,
content text,
PRIMARY KEY (tenant_id, id)
);
```
### Creating a GIN Index
`pg_bigm` supports full-text search indexes:
* gin must be used as an index method. GiST is not available for pg\_bigm.
* gin\_bigm\_ops must be used as an operator class.
```sql theme={null}
CREATE INDEX articles_content_idx ON articles USING gin (content gin_bigm_ops);
```
### Inserting Sample Data
```sql theme={null}
-- We need to create a tenant first
INSERT INTO tenants (id, name) VALUES
('d1c06023-3421-4fbb-9dd1-c96e42d2fd02', 'Tenant 1');
INSERT INTO articles (tenant_id, id, title, content) VALUES
('d1c06023-3421-4fbb-9dd1-c96e42d2fd02', 1, 'PostgreSQL Tutorial', 'Learn about PostgreSQL database management...'),
('d1c06023-3421-4fbb-9dd1-c96e42d2fd02', 2, 'Database Design', 'Best practices for designing databases...'),
('d1c06023-3421-4fbb-9dd1-c96e42d2fd02', 3, 'Query Optimization', 'Tips for optimizing database queries...');
```
### Search Examples
Simple partial matching:
```sql theme={null}
SELECT title, content
FROM articles
WHERE content LIKE '%database%';
```
Using similarity search:
```sql theme={null}
SELECT title, content, similarity(content, 'postgresql database') as sim
FROM articles
WHERE content % 'postgresql database'
ORDER BY sim DESC;
```
Finding similar strings:
```sql theme={null}
SELECT title
FROM articles
WHERE similarity(title, 'PostgeSQL') > 0.3; -- Will match 'PostgreSQL' despite typo
```
## Functions and Operators
### Main Functions
* `similarity(text, text)`: Returns similarity between two strings (0.0 to 1.0)
* `show_bigm(text)`: Shows all bigrams in a string
* `bigm_similarity_threshold`: Sets threshold for `%` operator (default: 0.3)
### Operators
* `LIKE`: Standard pattern matching
* `%`: Similarity search operator
* `=~`: Regular expression match with bigram index support
## Configuration Parameters
* `pg_bigm.similarity_threshold`: Default similarity threshold (0.0 to 1.0)
* `pg_bigm.enable_recheck`: Whether to recheck similarity in search results
* `pg_bigm.gin_key_limit`: Maximum number of bigrams for GIN index
## Use Cases
* Fuzzy text search
* Spell-check functionality
* Similar content matching
* Auto-complete suggestions
* Typo-tolerant search
* Multi-language text search
## Performance Considerations
* GIN indexes can be large, plan storage accordingly
* Index creation might be slow for large tables
* Index only necessary columns
* Monitor index size and search performance
* Adjust similarity threshold to balance precision and recall
* Consider using `pg_bigm.enable_recheck` for better accuracy
## Limitations
* Indexes can be larger compared to traditional B-tree indexes
* Not suitable for exact matching (use standard indexes instead)
* May require more memory during search operations
* Performance depends on similarity threshold and data size
## Additional Resources
* [pg\_bigm official repository](https://github.com/pgbigm/pg_bigm)
* [PostgreSQL Text Search Documentation](https://www.postgresql.org/docs/current/textsearch.html)
# pg_similarity
Source: https://thenile.dev/docs/extensions/pg_similarity
Text similarity measures in PostgreSQL
The `pg_similarity` extension provides a collection of similarity measures for comparing text strings in PostgreSQL databases.
It includes a comprehensive collection of search algorithms, which you can find listed toward the end of this document.
Your Nile database arrives with this extension already enabled.
## Usage Examples
Lets show how to use the `pg_similarity` extension to find similar products in our database.
### Creating a Table for Text Comparison
```sql theme={null}
CREATE TABLE products (
tenant_id uuid,
id integer,
name text,
description text,
PRIMARY KEY (tenant_id, id)
);
```
### Inserting Sample Data
```sql theme={null}
-- Create a tenant first
INSERT INTO tenants (id, name) VALUES
('d1c06023-3421-4fbb-9dd1-c96e42d2fd02', 'Tenant 1');
INSERT INTO products (tenant_id, id, name, description) VALUES
('d1c06023-3421-4fbb-9dd1-c96e42d2fd02', 1, 'Laptop Computer', 'High-performance laptop'),
('d1c06023-3421-4fbb-9dd1-c96e42d2fd02', 2, 'Laptop Computr', 'High-performance notebook'),
('d1c06023-3421-4fbb-9dd1-c96e42d2fd02', 3, 'Desktop Computer', 'Powerful desktop workstation');
```
### Finding Similar Names
```sql theme={null}
-- Using Levenshtein for typo detection
SELECT name,
lev(name, 'Laptop Computer') as similarity
FROM products
ORDER BY similarity DESC;
-- Using operator syntax
SELECT name
FROM products
WHERE name ~== 'Laptop Computer';
-- Using threshold to show only more similar results
SET pg_similarity.levenshtein_threshold = 0.8;
SELECT name
FROM products
WHERE name ~== 'Laptop Computer';
```
### Finding Similar Content
```sql theme={null}
-- Using Cosine for content matching
SELECT name,
cosine(description, 'high performance notebook') as similarity
FROM products
WHERE cosine(description, 'high performance notebook') > 0.5
ORDER BY similarity DESC;
-- Using operator syntax
-- This will return nothing because the threshold is 0.7
SELECT name
FROM products
WHERE description ~## 'high performance notebook';
-- Set threshold for specific algorithm
SET pg_similarity.cosine_threshold = 0.3;
-- Now this will return the results
SELECT name
FROM products
WHERE description ~## 'high performance notebook';
```
## Configuration
Each similarity measure has two or three configuration options:
* `threshold`: The threshold for the similarity measure.
* `is_normalized`: Whether the similarity measure is normalized (between 0 and 1) or not.
* `tokenizer`: The tokenizer to use for the similarity measure ( Default is alnum, and other options are gram, word, and camelcase). Note that not every algorithm supports the tokenizer option.
To use a specific configuration, you can use `SET pg_similarity._ = value`.
For example, to use the `cosine` similarity measure with a threshold of 0.3, you can use:
```sql theme={null}
SET pg_similarity.cosine_threshold = 0.3;
```
To reset the threshold to the default value, you can use:
```sql theme={null}
RESET pg_similarity.cosine_threshold;
```
## Common Use Cases
* Finding duplicate records with slight variations
* Implementing spell checking and typo-tolerant search
* Matching similar names or addresses
* Finding similar content
* Phonetic matching (e.g., "Smith" vs "Smyth")
* Record linkage across databases
## Complete Algorithm Reference
The extension includes the following similarity algorithms:
\| Algorithm | Function | Operator |
\| -------------------- | -------------------------------- | -------- | --- | --- |
\| Block | `block(text, text)` | `~++` |
\| Cosine | `cosine(text, text)` | `~##` |
\| Dice | `dice(text, text)` | `~-~` |
\| Euclidean | `euclidean(text, text)` | \~!! |
\| Hamming | `hamming(text, text)` | ~~@~~ |
\| Jaccard | `jaccard(text, text)` | `~??` |
\| Jaro | `jaro(text, text)` | `~%%` |
\| Jaro-Winkler | `jarowinkler(text, text)` | `~@@` |
\| Levenshtein | `lev(text, text)` | `~==` |
\| Matching | `matching(text, text)` | \~^^ |
\| Monge-Elkan | `mongeelkan(text, text)` | \~ | | |
\| Needleman-Wunsch | `needlemanwunch(text, text)` | ~~#~~ |
\| Overlap | `overlap(text, text)` | `~**` |
\| Q-Gram | `qgram(text, text)` | `~~~` |
\| Smith-Waterman | `smithwaterman(text, text)` | ~~=~~ |
\| Smith-Waterman-Gotoh | `smithwatermangotoh(text, text)` | ~~!~~ |
\| Soundex | `soundex(text, text)` | `~*~` |
Each algorithm is best suited for a different use case.
For example, `jaro` is better for name matching, while `levenshtein` is better for typo detection.
## Limitations
* Some algorithms may be computationally expensive
* Not all measures are suitable for all languages
* Memory usage can be high for large strings
* Some algorithms may not work well with very short strings
## Additional Resources
* [pg\_similarity repository with more complete documentation](https://github.com/eulerto/pg_similarity)
* [String Similarity Algorithms Overview](https://www.postgresql.org/docs/current/textsearch-intro.html)
# pg_trgm
Source: https://thenile.dev/docs/extensions/pg_trgm
Similarity search in PostgreSQL using trigrams
The `pg_trgm` extension provides trigram matching capabilities for fast text similarity search and fuzzy string matching in PostgreSQL. A trigram is a group of three consecutive characters taken from a string. This extension is particularly useful for implementing features like fuzzy search, spell checking, and finding similar strings.
Your Nile database arrives with the `pg_trgm` extension already enabled.
## Understanding Trigrams
A trigram is created by taking three consecutive characters from a text string. For example, the word "hello" contains these trigrams: " h" (with two spaces), "he", "hel", "ell", "llo", "lo " (with two spaces). The spaces at the beginning and end are important for matching word boundaries.
Similarity between strings is calculated by counting how many trigrams they share in common.
For example, "hello" and "helo" share most of their trigrams (like " h", "hel", "lo "),
resulting in a high similarity score. This makes trigrams excellent for fuzzy matching and finding similar strings even when they contain typos.
## Usage Examples
Let's explore how to use pg\_trgm with a practical example using a products database.
### Creating a Table with Text Search
```sql theme={null}
CREATE TABLE products (
tenant_id uuid,
id integer,
name text,
description text,
PRIMARY KEY (tenant_id, id)
);
-- Create a GiST index for faster similarity searches
CREATE INDEX trgm_idx_products_name ON products USING gist (name gist_trgm_ops);
-- Or create a GIN index (faster searches, slower updates, more space)
CREATE INDEX trgm_gin_idx_products_name ON products USING gin (name gin_trgm_ops);
```
### Inserting Sample Data
```sql theme={null}
-- Create a tenant first
INSERT INTO tenants (id, name) VALUES
('d1c06023-3421-4fbb-9dd1-c96e42d2fd02', 'Tenant 1');
INSERT INTO products (tenant_id, id, name, description) VALUES
('d1c06023-3421-4fbb-9dd1-c96e42d2fd02', 1, 'iPhone 13 Pro', 'Latest Apple smartphone'),
('d1c06023-3421-4fbb-9dd1-c96e42d2fd02', 2, 'iPhne 13', 'Budget Apple smartphone'),
('d1c06023-3421-4fbb-9dd1-c96e42d2fd02', 3, 'Samsung Galaxy S21', 'Android flagship phone');
```
### Basic Similarity Queries
```sql theme={null}
-- Find similarity between two strings (returns a number between 0 and 1)
SELECT similarity('iPhone', 'iPhne');
-- Find all products with names similar to 'iPhone' (overall similarity)
SELECT name, similarity(name, 'iPhone') AS sim
FROM products
WHERE name % 'iPhone' -- Uses similarity threshold
ORDER BY sim DESC;
-- Find products that contain something similar to 'phone' (substring matching)
SELECT name, similarity(name, 'iPhone') AS sim
FROM products
WHERE name %> 'phone'; -- Better for substring/pattern matching
```
### Show Trigrams
```sql theme={null}
-- Show trigrams in a string
SELECT show_trgm('iPhone');
```
### Setting Similarity Threshold
```sql theme={null}
-- Set the similarity threshold (default is 0.3)
SET pg_trgm.similarity_threshold = 0.3;
-- Query using the new threshold
SELECT name
FROM products
WHERE name % 'iPhone';
```
### Word Similarity
```sql theme={null}
-- Find word similarity (better for matching whole words)
SELECT name, word_similarity('iPhone', name) as sim
FROM products
ORDER BY sim DESC;
-- Using strict word similarity
SELECT name, strict_word_similarity('iPhone', name) as sim
FROM products
ORDER BY sim DESC;
```
## Functions and Operators
### Main Functions
* `similarity(text, text)`: Returns similarity between two strings (0 to 1)
* `show_trgm(text)`: Shows trigrams in a string
* `word_similarity(text, text)`: Returns word-based similarity
* `strict_word_similarity(text, text)`: Returns strict word-based similarity
* `show_limit()`: Shows current similarity threshold
### Operators
* `%`: Returns true if strings are similar (uses similarity threshold)
* `<%`: Returns true if first string is less similar than second
* `%>`: Returns true if first string is more similar than second
* `<->`: Returns distance between strings (1 - similarity)
* `<<->`: Returns word-based distance
* `<->>`: Returns strict word-based distance
## Index Types
pg\_trgm supports two types of indexes:
### GiST Index
```sql theme={null}
CREATE INDEX trgm_gist_idx ON table_name USING gist (column_name gist_trgm_ops);
```
* Balanced performance between search and update
* Smaller index size
* Good for dynamic data
### GIN Index
```sql theme={null}
CREATE INDEX trgm_gin_idx ON table_name USING gin (column_name gin_trgm_ops);
```
* Faster searches
* Slower updates
* Larger index size
* Better for static data
## Best Practices
1. **Index Selection**:
* Use GIN for mostly-read data
* Use GiST for frequently updated data
* Consider creating indexes only on frequently searched columns
2. **Threshold Tuning**:
* Lower threshold (e.g., 0.2) for more matches
* Higher threshold (e.g., 0.5) for stricter matching
* Test with your data to find the optimal value
3. **Performance Optimization**:
* Use word\_similarity() for whole word matching
* Create indexes on specific columns rather than all text columns
* Monitor index size and rebuild when necessary
## Common Use Cases
* Fuzzy search functionality
* Spell-check suggestions
* Auto-complete features
* Finding similar product names
* Matching addresses with typos
* Search with tolerance for misspellings
## Performance Considerations
* GIN indexes provide faster search but slower updates
* Index size can be large for text columns with many unique values
* Consider partial indexes for large tables
* Monitor and adjust similarity threshold based on false positive/negative rates
## Limitations
* Not suitable for very short strings (less than 3 characters)
* May produce false positives
* Index size can be large for big text columns
* Not ideal for exact matching (use standard indexes instead)
## Additional Resources
* [PostgreSQL pg\_trgm Documentation](https://www.postgresql.org/docs/current/pgtrgm.html)
* [Text Search in PostgreSQL](https://www.postgresql.org/docs/current/textsearch-indexes.html)
# Crypto
Source: https://thenile.dev/docs/extensions/pgcrypto
Cryptographic functions for PostgreSQL
The `pgcrypto` extension provides cryptographic functions for PostgreSQL, including hashing, encryption, and random data generation.
Your Nile database arrives with the pgcrypto extension already enabled.
## Overview
The pgcrypto extension provides functions for:
* Password hashing
* General-purpose hashing
* Encryption (symmetric and asymmetric)
* Random data generation
* Message signing and verification
## Password Hashing
### Using crypt()
The `crypt()` function is recommended for password hashing:
```sql theme={null}
-- Create a users table with hashed passwords
CREATE TABLE password_hashes (
tenant_id uuid,
user_id uuid DEFAULT uuid_generate_v4(),
email text,
password_hash text,
PRIMARY KEY (tenant_id, user_id)
);
-- Insert a user with a hashed password (using blowfish)
INSERT INTO password_hashes (tenant_id, email, password_hash) VALUES
('11111111-1111-1111-1111-111111111111',
'user@example.com',
public.crypt('user_password', public.gen_salt('bf')));
-- Verify password
SELECT user_id
FROM password_hashes
WHERE email = 'user@example.com'
AND password_hash = public.crypt('user_password', password_hash);
```
### Symmetric Encryption
```sql theme={null}
-- Create a table with encrypted data
CREATE TABLE sensitive_data (
tenant_id uuid,
record_id uuid DEFAULT uuid_generate_v4(),
description text,
encrypted_data bytea,
PRIMARY KEY (tenant_id, record_id)
);
-- Insert encrypted data (AES-128-CBC)
INSERT INTO sensitive_data (tenant_id, description, encrypted_data) VALUES
('11111111-1111-1111-1111-111111111111',
'Credit Card',
public.encrypt(
'4111111111111111'::bytea,
'encryption_key',
'aes'
));
-- Decrypt data
SELECT description,
convert_from(
public.decrypt(
encrypted_data,
'encryption_key',
'aes'
),
'utf-8'
) as decrypted_data
FROM sensitive_data;
```
## Random Data Generation
```sql theme={null}
-- Generate random bytes
SELECT public.gen_random_bytes(16); -- 16 random bytes
-- Generate random UUID (alternative to uuid-ossp)
SELECT public.gen_random_uuid(); -- Random UUID
```
## Dos and Don'ts
### Password Storage
✅ Use crypt() with Blowfish:
`password_hash = public.crypt(password, public.gen_salt('bf', 8))`
❌ Don't store plain MD5 (unsafe!):
`password_hash = public.md5(password)`
### Encryption Key Management
✅ Store keys securely outside the database: `encrypted_data = public.encrypt(data, current_setting('app.encryption_key'), 'aes')`
❌ Don't store keys in the database
❌ Don't hardcode keys in application code
### Salt Generation
✅ Generate a new salt for each password: `SELECT public.gen_salt('bf', 8);`
❌ Don't reuse salts
❌ Don't use static salts
## Performance Considerations
1. Hashing and encryption are CPU-intensive operations. Consider caching results when appropriate.
2. Encrypted columns cannot be effectively indexed. Consider indexing non-sensitive fields instead.
## Additional Resources
* [PostgreSQL pgcrypto Documentation](https://www.postgresql.org/docs/current/pgcrypto.html)
* [OpenPGP Message Format](https://www.rfc-editor.org/rfc/rfc4880)
* [Password Hashing Best Practices](https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html)
# pgRouting
Source: https://thenile.dev/docs/extensions/pgrouting
Geospatial routing extension for PostgreSQL
The `pgRouting` extension extends PostgreSQL and PostGIS to provide geospatial routing and network analysis functionality. It enables you to perform shortest path calculations, traveling salesperson solutions, and other routing operations on spatial networks like road networks.
Your Nile database arrives with the `pgRouting` extension and its dependency `postgis` already enabled.
## Understanding pgRouting
pgRouting works with network topologies stored in PostgreSQL/PostGIS. A network topology consists of:
* Vertices (nodes/intersections)
* Edges (segments/roads)
* Costs (distance, time, or other metrics)
### Key Features
* **Multiple Routing Algorithms**: Dijkstra, A\*, Traveling Salesperson Problem (TSP)
* **Flexible Cost Calculations**: Support for distance, time, and custom cost functions
* **Turn Restriction Support**: Handle real-world routing constraints
* **Dynamic Cost Updates**: Modify costs based on traffic or other conditions
* **Large Network Support**: Efficient handling of large road networks
## Usage Examples
Let's create a simple road network and perform various routing operations.
### Creating the Network Table
```sql theme={null}
-- Create a table for our road network
CREATE TABLE roads (
id BIGSERIAL PRIMARY KEY,
name VARCHAR(100),
source BIGINT, -- Pre-assigned source node ID
target BIGINT, -- Pre-assigned target node ID
cost FLOAT,
reverse_cost FLOAT,
length_m FLOAT,
the_geom geometry
);
-- Create indices for better performance
CREATE INDEX roads_source_idx ON roads(source);
CREATE INDEX roads_target_idx ON roads(target);
CREATE INDEX roads_geom_idx ON roads USING GIST(the_geom);
```
### Inserting Sample Data
```sql theme={null}
-- Insert sample road segments with pre-defined source/target nodes
INSERT INTO roads (name, the_geom, length_m, cost, reverse_cost, source, target) VALUES
('Main St',
ST_GeomFromText('LINESTRING(-122.678 45.526, -122.675 45.526)', 4326),
300, 300, 300, 1, 2),
('Oak Ave',
ST_GeomFromText('LINESTRING(-122.675 45.526, -122.675 45.524)', 4326),
200, 200, 200, 2, 3),
('Pine St',
ST_GeomFromText('LINESTRING(-122.675 45.524, -122.678 45.524)', 4326),
300, 300, 300, 3, 4);
```
### Basic Shortest Path Query
Using Dijkstra's algorithm:
```sql theme={null}
-- Find shortest path between two points
SELECT seq, node, edge, cost
FROM pgr_dijkstra(
'SELECT id, source, target, cost FROM roads',
1, -- starting vertex
3, -- ending vertex
directed := false
);
```
### A\* Search
When you have geographic coordinates:
```sql theme={null}
SELECT seq, node, edge, cost
FROM pgr_astar(
'SELECT id, source, target, cost,
ST_X(ST_StartPoint(the_geom)) AS x1,
ST_Y(ST_StartPoint(the_geom)) AS y1,
ST_X(ST_EndPoint(the_geom)) AS x2,
ST_Y(ST_EndPoint(the_geom)) AS y2
FROM roads',
1, -- starting vertex
3, -- ending vertex
directed := false
);
```
### Driving Distance
Find all reachable nodes within a certain cost:
```sql theme={null}
SELECT node, edge, cost
FROM pgr_drivingDistance(
'SELECT id, source, target, cost FROM roads',
1, -- starting vertex
1000, -- maximum cost
directed := false
);
```
## Common Use Cases
* Navigation systems
* Delivery route optimization
* Service area analysis
* Emergency response planning
* Public transport routing
* Traffic impact analysis
## Limitations
* Memory usage increases with network size
* Some algorithms have exponential complexity
* Real-time updates can be challenging
* Turn restrictions increase complexity
* Limited support for time-dependent routing
## Additional Resources
* [pgRouting Documentation](https://docs.pgrouting.org/)
* [pgRouting Workshop](https://workshop.pgrouting.org/)
* [PostGIS Documentation](https://postgis.net/documentation/)
* [OpenStreetMap Integration](https://wiki.openstreetmap.org/wiki/pgRouting)
# Pgvectorscale
Source: https://thenile.dev/docs/extensions/pgvectorscale
DiskANN index support for pgvector
The `pgvectorscale` extension adds diskANN index support for pgvector.
This extension is useful in cases where `pgvector`'s `hnsw` index does not fit into available memory and as a result the ANN search does not perform as expected.
## Key Features
* StreamingDiskANN index - disk-backed HNSW variant.
* Statistical Binary Quantization (SBQ)
* Label-based filtering combined with DiskANN index.
## Example: DiskANN index on shared table
To keep the example readable we'll work with **3-dimensional vectors**.
Swap `VECTOR(3)` for `VECTOR(768)` or `VECTOR(1536)` in real apps.
```sql theme={null}
-- 1. Shared data table
CREATE TABLE document_embedding (
id BIGSERIAL PRIMARY KEY,
contents TEXT,
metadata JSONB,
embedding VECTOR(3)
);
-- 2. Seed with tiny sample data
INSERT INTO document_embedding (contents, metadata, embedding) VALUES
('T-shirt', '{"category":"apparel"}', '[0.10, 0.20, 0.30]'),
('Sweater', '{"category":"apparel"}', '[0.12, 0.18, 0.33]'),
('Coffee mug', '{"category":"kitchen"}', '[0.90, 0.80, 0.70]');
-- 3. Build a DiskANN index (cosine distance)
CREATE INDEX document_embedding_diskann_idx
ON document_embedding
USING diskann (embedding vector_cosine_ops);
-- 4. k-NN query (top-2 similar items)
SELECT id, contents, metadata
FROM document_embedding
ORDER BY embedding <=> '[0.11, 0.21, 0.29]' -- query vector
LIMIT 2;
```
You should see the two apparel rows first - a good sanity check that the index works.
## Example: DiskANN index on tenant-aware table
```sql theme={null}
-- 1. Tenant-aware table
CREATE TABLE tenant_embedding (
tenant_id UUID NOT NULL,
doc_id BIGINT,
embedding VECTOR(2), -- using tiny 2‑dim vectors for demo
metadata JSONB,
PRIMARY KEY (tenant_id, doc_id)
);
-- 2. Create some tenants
INSERT INTO tenants (id, name) VALUES
('11111111-1111-1111-1111-111111111111', 'Tenant A');
INSERT INTO tenants (id, name) VALUES
('22222222-2222-2222-2222-222222222222', 'Tenant B');
-- 3. Seed soome data
INSERT INTO tenant_embedding (tenant_id, doc_id, embedding, metadata) VALUES
('11111111-1111-1111-1111-111111111111', 1, '[0.05, 0.95]', '{"title":"DocA"}'),
('11111111-1111-1111-1111-111111111111', 2, '[0.04, 0.90]', '{"title":"DocB"}');
INSERT INTO tenant_embedding (tenant_id, doc_id, embedding, metadata) VALUES
('22222222-2222-2222-2222-222222222222', 1, '[0.80, 0.20]', '{"title":"DocC"}');
-- 3. Create an index (Nile will partition by tenant_id)
CREATE INDEX tenant_embedding_diskann_idx
ON tenant_embedding
USING diskann (embedding vector_cosine_ops);
-- 4. Tenant‑scoped ANN query
SET nile.tenant_id = '11111111-1111-1111-1111-111111111111';
SELECT doc_id, metadata
FROM tenant_embedding
ORDER BY embedding <=> '[0.06, 0.92]'
LIMIT 2;
```
## Example: Label-based filtering
Label-based filtering is a technique that allows you to filter the results of an ANN search based on a label while using the DiskANN index.
Other filters are supported, but will use pgvector's post-filtering (i.e. after the ANN search).
In order to use label based filtering, you need to:
* Create a label column in your table. It has to be an array of `smallint`s. Other types will revert to using the post-filtering.
* Create a diskann index that uses both the embedding and the label column.
* Use the `&&` (array intersection) operator in search queries.
* Optional, but recommended: Use a separate table and joins to translate smallint labels to meaningful descriptions.
```sql theme={null}
-- 1. Create a label column
CREATE TABLE documents (
id BIGSERIAL PRIMARY KEY,
embedding VECTOR(3),
labels SMALLINT[]
);
-- 2. Create an index on the label column
-- Insert a couple of demo rows
INSERT INTO documents (embedding, labels) VALUES
('[0.3,0.2,0.1]', ARRAY[1]), -- label 1 = science
('[0.35,0.25,0.05]', ARRAY[1,2]), -- label 2 = business
('[0.9,0.8,0.7]', ARRAY[3]); -- label 3 = art
-- 3. Create an index on the label column
CREATE INDEX documents_ann_idx
ON documents
USING diskann (embedding vector_cosine_ops, labels);
-- 4. Query with label-based filtering
SELECT *
FROM documents
WHERE labels && ARRAY[1,2]
ORDER BY embedding <=> '[0.32,0.18,0.12]'
LIMIT 5;
-- 5. Optional: Translate labels to descriptions
CREATE TABLE labels (
id SMALLINT PRIMARY KEY,
description TEXT
);
INSERT INTO labels (id, description) VALUES
(1, 'Science'),
(2, 'Business'),
(3, 'Art');
-- 6. Query with label-based filtering and description
SELECT d.*
FROM documents d
WHERE d.labels && (
SELECT array_agg(id)
FROM labels
WHERE description in ('Science', 'Business')
)
ORDER BY d.embedding <=> '[0.32,0.18,0.12]'
LIMIT 5;
```
## Limitations
* DiskANN index supports `cosine`, `l2` and `inner_product` distance metrics, not the entire pgvector's set of distance metrics.
* Label-based filtering is only supported for `smallint` arrays and the `&&` operator. Other types will revert to using the post-filtering.
* DiskANN is best suited for datasets where `hnsw` index would be too large to fit into memory. For smaller datasets, `hnsw` is still a good choice.
## Additional Resources
[Pgvectorscale github repository](https://github.com/timescale/pgvectorscale)
# PostGIS
Source: https://thenile.dev/docs/extensions/postgis
Spatial and Geographic Objects for PostgreSQL
The `PostGIS` extension adds support for geographic objects to PostgreSQL, allowing you to store, query, and manipulate spatial data. It effectively turns PostgreSQL into a spatial database.
Your Nile database arrives with the `PostGIS` extension already enabled.
## Quick Start
Let's walk through some common PostGIS operations using a simple example of storing and querying location data.
### Creating a Spatial Table
```sql theme={null}
-- Create a table for storing points of interest
CREATE TABLE points_of_interest (
tenant_id uuid,
id INTEGER,
name VARCHAR(100),
type VARCHAR(50),
-- POINT geometry in WGS84 (latitude/longitude)
location geometry(POINT, 4326),
PRIMARY KEY (tenant_id, id)
);
-- Create a spatial index
CREATE INDEX points_of_interest_gist ON points_of_interest USING GIST(location);
```
### Inserting Data
```sql theme={null}
-- Create a tenant first
INSERT INTO tenants (id, name) VALUES
('d1c06023-3421-4fbb-9dd1-c96e42d2fd02', 'Tenant 1');
-- Insert some points of interest
INSERT INTO points_of_interest (tenant_id, id, name, type, location) VALUES
('d1c06023-3421-4fbb-9dd1-c96e42d2fd02', 1, 'Central Park', 'park',
ST_SetSRID(ST_MakePoint(-73.965355, 40.782865), 4326)),
('d1c06023-3421-4fbb-9dd1-c96e42d2fd02', 2, 'Empire State', 'building',
ST_SetSRID(ST_MakePoint(-73.985428, 40.748817), 4326)),
('d1c06023-3421-4fbb-9dd1-c96e42d2fd02', 3, 'Statue of Liberty', 'monument',
ST_SetSRID(ST_MakePoint(-74.044502, 40.689247), 4326));
```
### Basic Spatial Queries
Find all points within 5km of a location:
```sql theme={null}
SELECT name,
ST_Distance(
location::geography,
ST_SetSRID(ST_MakePoint(-73.985428, 40.748817), 4326)::geography
) as distance_meters
FROM points_of_interest
WHERE ST_DWithin(
location::geography,
ST_SetSRID(ST_MakePoint(-73.985428, 40.748817), 4326)::geography,
5000 -- 5km in meters
)
ORDER BY distance_meters;
```
Calculate distance between two points:
```sql theme={null}
-- Distance calculations default to meters, you can multiple by 0.000621371 to get miles
SELECT ST_Distance(
(SELECT location::geography FROM points_of_interest WHERE name = 'Central Park'),
(SELECT location::geography FROM points_of_interest WHERE name = 'Empire State')
) as distance_meters;
```
### Working with Areas
Create and query polygons:
```sql theme={null}
-- Create a table for areas
CREATE TABLE areas (
tenant_id uuid,
id INTEGER,
name VARCHAR(100),
boundary geometry(POLYGON, 4326),
PRIMARY KEY (tenant_id, id)
);
-- Insert multiple polygons (simplified boundaries)
INSERT INTO areas (tenant_id, id, name, boundary) VALUES
('d1c06023-3421-4fbb-9dd1-c96e42d2fd02', 1, 'Central Park',
ST_SetSRID(ST_MakePolygon(ST_GeomFromText('LINESTRING(
-73.968285 40.785091,
-73.961675 40.785091,
-73.961675 40.780467,
-73.968285 40.780467,
-73.968285 40.785091
)')), 4326)),
('d1c06023-3421-4fbb-9dd1-c96e42d2fd02', 2, 'Area1',
ST_SetSRID(ST_MakePolygon(ST_GeomFromText('LINESTRING(
-73.965 40.783,
-73.960 40.783,
-73.960 40.779,
-73.965 40.779,
-73.965 40.783
)')), 4326)),
('d1c06023-3421-4fbb-9dd1-c96e42d2fd02', 3, 'Area2',
ST_SetSRID(ST_MakePolygon(ST_GeomFromText('LINESTRING(
-73.963 40.784,
-73.958 40.784,
-73.958 40.780,
-73.963 40.780,
-73.963 40.784
)')), 4326));
-- Find points within the area
SELECT p.name
FROM points_of_interest p
JOIN areas a ON ST_Contains(a.boundary, p.location)
WHERE a.name = 'Central Park';
```
## Common Operations
### Coordinate Transformations
Convert between coordinate systems:
```sql theme={null}
-- Convert from WGS84 (EPSG:4326) to Web Mercator (EPSG:3857)
SELECT ST_Transform(
ST_SetSRID(ST_MakePoint(-73.985428, 40.748817), 4326),
3857
);
```
### Distance Calculations
```sql theme={null}
-- Calculate distance in meters
SELECT ST_Distance(
ST_SetSRID(ST_MakePoint(-73.985428, 40.748817), 4326)::geography,
ST_SetSRID(ST_MakePoint(-73.968285, 40.785091), 4326)::geography
);
```
### Spatial Relationships
```sql theme={null}
-- Check if point is within polygon
SELECT ST_Contains(
(SELECT boundary FROM areas WHERE name = 'Central Park'),
(SELECT location FROM points_of_interest WHERE name = 'Empire State')
);
-- Find intersection of two polygons
SELECT ST_Intersection(a.boundary, b.boundary)
FROM areas a, areas b
WHERE a.name = 'Area1' AND b.name = 'Area2';
```
### Geometry Creation
```sql theme={null}
-- Create a point
SELECT ST_SetSRID(ST_MakePoint(-73.985428, 40.748817), 4326);
-- Create a line
SELECT ST_MakeLine(
ST_MakePoint(-73.985428, 40.748817),
ST_MakePoint(-73.968285, 40.785091)
);
-- Create a polygon
SELECT ST_MakePolygon(ST_GeomFromText('LINESTRING(
0 0, 1 0, 1 1, 0 1, 0 0
)'));
```
## Best Practices
1. **Indexing**:
* Always create spatial indexes (GiST) on geometry columns
* Use appropriate coordinate systems for your use case
2. **Performance**:
* Use ST\_DWithin instead of ST\_Distance for radius searches
* Cast to geography type for accurate earth-distance calculations
* Consider clustering on spatial indexes for large datasets
3. **Data Quality**:
* Validate geometries using ST\_IsValid
* Use appropriate SRID for your data
* Clean up invalid geometries using ST\_MakeValid
## Common Use Cases
* Location-based services
* Geofencing
* Territory management
* Asset tracking
* Spatial analysis
* Map visualization
* Route planning
* Environmental analysis
## Additional Resources
* [PostGIS Documentation](https://postgis.net/documentation/)
* [PostGIS Introduction](https://postgis.net/workshops/postgis-intro/)
* [Coordinate Systems Guide](https://postgis.net/workshops/postgis-intro/projection.html)
* [PostGIS in Action Book](https://www.manning.com/books/postgis-in-action-third-edition)
# PostGIS Raster
Source: https://thenile.dev/docs/extensions/postgis_raster
Raster Data Support for PostGIS
The `postgis_raster` extension adds support for raster (grid-based) spatial data to PostGIS. It enables you to store, analyze, and process raster data such as satellite imagery, elevation models, and other gridded datasets directly in your PostgreSQL database.
Your Nile database arrives with the `postgis_raster` extension and its dependency `postgis` already enabled.
## Understanding Raster Data
A raster consists of a matrix of cells (pixels) organized into rows and columns where each cell contains a value representing information such as:
* Elevation data (DEM - Digital Elevation Model)
* Satellite imagery
* Temperature maps
* Land use classification
* Any other grid-based spatial data
A raster can store multiple layers of data, each layer is called a band. For example a band can
represent elevation data, another band can represent temperature data. In satellite imagery, each band
typically represents a different wavelength of light (Red, Green, Blue).
## Quick Start
Let's walk through some common operations with raster data.
### Creating a Raster Table
```sql theme={null}
-- Create a table for storing elevation data
CREATE TABLE elevation_data (
id INTEGER PRIMARY KEY,
name VARCHAR(100),
acquisition_date DATE,
-- Raster column with spatial reference system EPSG:4326
rast raster
);
```
### Loading Raster Data
```sql theme={null}
-- Insert a simple 3x3 raster representing elevation data
INSERT INTO elevation_data (id, name, acquisition_date, rast)
SELECT
1,
'Sample DEM',
'2024-01-01',
ST_AddBand(
ST_MakeEmptyRaster(3, 3, -74.0, 41.0, 0.01, -0.01, 0, 0, 4326),
'32BF'::text, -- 32-bit float
1.0, 0.0 -- pixel value 1.0, nodata value 0.0
);
-- Update raster values (example elevation data in meters)
UPDATE elevation_data
SET rast = ST_SetValues(
rast,
1, -- band number
1, 1, -- starting pixel
ARRAY[
[100.0, 120.0, 130.0],
[110.0, 125.0, 135.0],
[105.0, 115.0, 140.0]
]::double precision[][]
)
WHERE id = 1;
```
### Basic Raster Operations
Query pixel values at a specific point:
```sql theme={null}
-- Get elevation at a specific coordinate
SELECT ST_Value(rast, 1, ST_SetSRID(ST_MakePoint(-74.0, 41.0), 4326))
FROM elevation_data
WHERE id = 1;
```
Calculate statistics for a raster:
```sql theme={null}
-- Get summary statistics for the raster
SELECT
ST_SummaryStats(rast) AS stats,
(ST_SummaryStats(rast)).min AS min_elevation,
(ST_SummaryStats(rast)).max AS max_elevation,
(ST_SummaryStats(rast)).mean AS mean_elevation
FROM elevation_data
WHERE id = 1;
```
Resample a raster to different resolution:
```sql theme={null}
-- Resample to 50% of original resolution
SELECT ST_Resample(
rast,
scalex := 0.02, -- New pixel width
scaley := -0.02, -- New pixel height
algorithm := 'NearestNeighbor' -- Resampling algorithm
)
FROM elevation_data
WHERE id = 1;
```
### Raster Analysis
Calculate slope from elevation data:
```sql theme={null}
-- Generate slope (in degrees) from elevation raster
-- Returns a new raster where pixel values represent slope angles
-- Use ST_SummaryStats() to view actual degree values
SELECT ST_Slope(
rast,
1, -- Band number
'32BF', -- Pixel type
'DEGREE' -- Output units
)
FROM elevation_data
WHERE id = 1;
```
Generate contour lines from elevation data
```sql theme={null}
-- Generate contour lines from elevation data
-- Returns a MultiLineString geometry where each line represents locations of equal elevation
-- Interval of 10 meters means lines will be drawn at 100m, 110m, 120m elevation etc.
SELECT ST_Contour(
rast,
1, -- Band number
10.0 -- Contour interval
)
FROM elevation_data
WHERE id = 1;
```
### Raster Properties
```sql theme={null}
-- Get raster metadata
SELECT
ST_Width(rast) AS width,
ST_Height(rast) AS height,
ST_NumBands(rast) AS num_bands,
ST_PixelWidth(rast) AS pixel_width,
ST_PixelHeight(rast) AS pixel_height,
ST_SRID(rast) AS srid
FROM elevation_data
WHERE id = 1;
```
### Raster Manipulation
```sql theme={null}
-- Clip raster to polygon
SELECT ST_Clip(
rast,
ST_GeomFromText('POLYGON((-74.01 41.01, -73.99 41.01, -73.99 40.99, -74.01 40.99, -74.01 41.01))', 4326)
)
FROM elevation_data
WHERE id = 1;
-- Reproject raster to different coordinate system
SELECT ST_Transform(rast, 3857)
FROM elevation_data
WHERE id = 1;
```
## Best Practices
1. **Storage and Indexing**:
* Use appropriate pixel types for your data
* Create spatial indexes on raster columns
* Consider tiling large rasters
2. **Performance**:
* Use appropriate chunk sizes for large rasters
* Optimize raster resolution for your use case
* Consider using out-db raster storage for very large datasets
3. **Data Quality**:
* Validate raster data before loading
* Handle NODATA values appropriately
* Use appropriate resampling methods
## Common Use Cases
* Digital Elevation Models (DEM)
* Satellite imagery analysis
* Land use/land cover mapping
* Temperature and climate modeling
* Watershed analysis
* Viewshed analysis
* Terrain analysis
* Environmental monitoring
## Limitations
* Large raster datasets can consume significant storage
* Processing time increases with raster size
* Memory usage can be high for large raster operations
## Additional Resources
* [Introduction to Rasters in PostGIS](https://postgis.net/workshops/de/postgis-intro/rasters.html)
* [PostGIS Raster Reference](https://postgis.net/docs/RT_reference.html)
# Prefix
Source: https://thenile.dev/docs/extensions/prefix
Prefix search functionality for PostgreSQL
The `prefix` extension provides efficient prefix (starts-with) search functionality in PostgreSQL.
While it's useful for implementing autocomplete features and prefix-based filtering, it's particularly
important in telephony applications where call routing and costs depend on matching phone numbers
to operator prefixes.
Your Nile database arrives with the `prefix` extension already enabled.
## Understanding Prefix Search
A prefix search finds strings that begin with a specific pattern. In telephony applications,
this is crucial for:
* Matching phone numbers to carrier prefixes
* Determining call routing paths
* Calculating call costs based on destination
* Identifying geographic regions from area codes
For example:
* "1212" is a prefix for all London phone numbers starting with "1212"
* "+1" is a prefix for North American numbers
* "91" is a prefix for calls to India
## Quick Start
Let's walk through some common prefix search operations using telephony examples.
### Creating a Table for Phone Prefixes
```sql theme={null}
-- Create a table for phone number prefixes
CREATE TABLE phone_prefixes (
id INTEGER PRIMARY KEY,
prefix prefix_range,
carrier TEXT,
region TEXT,
rate_per_minute DECIMAL(10,4)
);
-- Create a prefix index on the prefix column
CREATE INDEX idx_prefix ON phone_prefixes USING gist(prefix);
```
### Inserting Sample Data
```sql theme={null}
-- Insert sample phone prefixes
INSERT INTO phone_prefixes (id, prefix, carrier, region, rate_per_minute) VALUES
(1, '1212', 'UK Telecom', 'London', 0.02),
(2, '1', 'US Carrier', 'North America', 0.01),
(3, '91', 'India Tel', 'India', 0.05),
(4, '44', 'UK Mobile', 'United Kingdom', 0.03),
(5, '86', 'China Tel', 'China', 0.04),
(6, '972', 'Israel Tel', 'Israel', 0.06);
```
### Basic Phone Number Queries
Find carrier and rate for a specific phone number:
```sql theme={null}
-- Find the matching prefix for a phone number
-- You should see that '1212' and 'UK Telecom' are returned
SELECT prefix, carrier, rate_per_minute
FROM phone_prefixes
WHERE prefix @> '12125551234'
ORDER BY length(prefix) DESC
LIMIT 1;
```
### Call Cost Calculations
```sql theme={null}
-- Create a table for call records
CREATE TABLE call_records (
tenant_id uuid,
id INTEGER,
caller_number TEXT,
callee_number TEXT,
duration_minutes INTEGER,
timestamp TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, id)
);
-- Insert sample call records
INSERT INTO call_records (tenant_id, id, caller_number, callee_number, duration_minutes) VALUES
('123e4567-e89b-12d3-a456-426614174000', 1, '12125550123', '912234567890', 5),
('123e4567-e89b-12d3-a456-426614174000', 2, '14155550123', '442087654321', 3),
('123e4567-e89b-12d3-a456-426614174000', 3, '16175550123', '861234567890', 8);
-- Example query to calculate call costs directly
SELECT
cr.callee_number,
cr.duration_minutes,
(cr.duration_minutes * pp.rate_per_minute) as cost
FROM call_records cr
LEFT JOIN LATERAL (
SELECT rate_per_minute
FROM phone_prefixes
WHERE prefix @> cr.callee_number
ORDER BY length(prefix) DESC
LIMIT 1
) pp ON true;
```
### Call Router Implementation
```sql theme={null}
-- Create a table for routing rules
CREATE TABLE routing_rules (
tenant_id uuid,
id INTEGER,
prefix prefix_range,
route_to TEXT,
priority INTEGER,
PRIMARY KEY (tenant_id, id)
);
-- Create prefix index
CREATE INDEX idx_routing ON routing_rules USING gist(prefix);
-- Insert sample routing rules
INSERT INTO routing_rules (tenant_id, id, prefix, route_to, priority) VALUES
('123e4567-e89b-12d3-a456-426614174000', 1, '1212', 'london-gateway-1', 100),
('123e4567-e89b-12d3-a456-426614174000', 2, '1', 'us-gateway-main', 50),
('123e4567-e89b-12d3-a456-426614174000', 3, '91', 'india-gateway-1', 75),
('123e4567-e89b-12d3-a456-426614174000', 4, '44', 'uk-gateway-main', 80),
('123e4567-e89b-12d3-a456-426614174000', 5, '86', 'china-gateway-1', 70);
-- Find route for a number
SELECT route_to
FROM routing_rules
WHERE prefix @> '12125551234'
ORDER BY priority DESC, length(prefix) DESC
LIMIT 1;
```
## Operators
The `prefix` extension introduces operators on prefix ranges:
* `@>`: Checks if the prefix range contains the given number or range
* `<@`: Checks if the prefix range or numberis contained within another prefix range
* `&&`: Checks for overlapping prefix ranges
* `|`: Is union of two prefix ranges
* `&`: Is intersection of two prefix ranges
In addition, operators `<=`, `>=`, `<`, `>`, `=`, `<>` are supported for prefix ranges.
## Common Use Cases
* Telephone number routing
* Call cost calculation
* Carrier and geographic prefix matching
* Product SKU searches (e.g., finding all products starting with 'ELEC-')
* URL path routing and matching
* File system path lookups
## Additional Resources
* [Prefix Extension Repository](https://github.com/dimitri/prefix)
* [PostgreSQL Pattern Matching](https://www.postgresql.org/docs/current/functions-matching.html)
# Quantile
Source: https://thenile.dev/docs/extensions/quantile
Efficient quantile and percentile calculations in PostgreSQL
The `quantile` extension provides efficient computation of quantiles and percentiles in PostgreSQL. It's particularly useful for statistical analysis, performance monitoring, and data distribution understanding.
Your Nile database arrives with the `quantile` extension already enabled.
## Understanding Quantiles
A quantile divides a dataset into equal-sized groups. Common examples include:
* Median (50th percentile)
* Quartiles (25th, 50th, 75th percentiles)
* Percentiles (dividing data into 100 groups)
* Custom quantiles (any division between 0 and 1)
## Quick Start
Let's explore quantile calculations with practical examples.
### Creating a Table with Sample Data
```sql theme={null}
-- Create a table for response times
CREATE TABLE api_responses (
tenant_id uuid,
id INTEGER,
endpoint TEXT,
response_time_ms INTEGER,
timestamp TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, id)
);
-- Create an index on response time for better performance
CREATE INDEX idx_response_time ON api_responses(response_time_ms);
```
```sql theme={null}
-- Create a tenant first
INSERT INTO tenants (id, name) VALUES
('d1c06023-3421-4fbb-9dd1-c96e42d2fd02', 'Tenant 1');
-- Insert sample response times across different hours
INSERT INTO api_responses (tenant_id, id, endpoint, response_time_ms, timestamp) VALUES
-- Data for current hour
('d1c06023-3421-4fbb-9dd1-c96e42d2fd02', 1, '/api/users', 45, CURRENT_TIMESTAMP),
('d1c06023-3421-4fbb-9dd1-c96e42d2fd02', 2, '/api/users', 52, CURRENT_TIMESTAMP),
('d1c06023-3421-4fbb-9dd1-c96e42d2fd02', 3, '/api/users', 138, CURRENT_TIMESTAMP),
-- Data from 1 hour ago
('d1c06023-3421-4fbb-9dd1-c96e42d2fd02', 4, '/api/users', 42, CURRENT_TIMESTAMP - INTERVAL '1 hour'),
('d1c06023-3421-4fbb-9dd1-c96e42d2fd02', 5, '/api/users', 58, CURRENT_TIMESTAMP - INTERVAL '1 hour'),
('d1c06023-3421-4fbb-9dd1-c96e42d2fd02', 6, '/api/users', 95, CURRENT_TIMESTAMP - INTERVAL '1 hour'),
-- Data from 2 hours ago
('d1c06023-3421-4fbb-9dd1-c96e42d2fd02', 7, '/api/orders', 123, CURRENT_TIMESTAMP - INTERVAL '2 hours'),
('d1c06023-3421-4fbb-9dd1-c96e42d2fd02', 8, '/api/orders', 95, CURRENT_TIMESTAMP - INTERVAL '2 hours'),
('d1c06023-3421-4fbb-9dd1-c96e42d2fd02', 9, '/api/orders', 167, CURRENT_TIMESTAMP - INTERVAL '2 hours'),
-- Data from 3 hours ago
('d1c06023-3421-4fbb-9dd1-c96e42d2fd02', 10, '/api/products', 67, CURRENT_TIMESTAMP - INTERVAL '3 hours'),
('d1c06023-3421-4fbb-9dd1-c96e42d2fd02', 11, '/api/products', 72, CURRENT_TIMESTAMP - INTERVAL '3 hours'),
('d1c06023-3421-4fbb-9dd1-c96e42d2fd02', 12, '/api/products', 158, CURRENT_TIMESTAMP - INTERVAL '3 hours');
```
### Basic Quantile Calculations
Calculate median response time:
```sql theme={null}
SELECT quantile(response_time_ms, 0.5) as median_response_time
FROM api_responses;
```
Calculate multiple percentiles:
```sql theme={null}
SELECT
endpoint,
quantile(response_time_ms, 0.5) as p50,
quantile(response_time_ms, 0.90) as p90,
quantile(response_time_ms, 0.95) as p95,
quantile(response_time_ms, 0.99) as p99
FROM api_responses
GROUP BY endpoint;
```
### Rolling Percentiles Example
```sql theme={null}
-- Calculate rolling percentiles for API response times
SELECT
endpoint as service,
date_trunc('hour', timestamp) as hour,
quantile(response_time_ms, ARRAY[0.5, 0.90, 0.95, 0.99]) as percentiles
FROM api_responses
WHERE timestamp >= NOW() - INTERVAL '24 hours'
GROUP BY endpoint, date_trunc('hour', timestamp)
ORDER BY service, hour;
```
This query will show you the 50th, 90th, 95th, and 99th percentiles of response times for each API endpoint, grouped by hour over the last 24 hours.
## Common Use Cases
1. **Performance Monitoring**
* Response time percentiles
* Resource usage distribution
* SLA compliance monitoring
2. **Financial Analysis**
* Price distribution analysis
* Risk assessment
* Portfolio performance metrics
3. **Quality Control**
* Process variation monitoring
* Outlier detection
* Manufacturing tolerances
## Additional Resources
* [PostgreSQL Aggregate Functions](https://www.postgresql.org/docs/current/functions-aggregate.html)
* [Statistical Functions in PostgreSQL](https://www.postgresql.org/docs/current/functions-aggregate.html#FUNCTIONS-AGGREGATE-STATISTICS-TABLE)
* [Time-Series Analysis in PostgreSQL](https://www.postgresql.org/docs/current/functions-aggregate.html#FUNCTIONS-AGGREGATE-WINDOW)
# Random
Source: https://thenile.dev/docs/extensions/random
Random data generator extension for PostgreSQL with wide range of data types
The `random` extension provides a collection of functions to generate random values for various data types in PostgreSQL. It's particularly useful for testing, data generation, and creating reproducible datasets.
Your Nile database arrives with the `random` extension already enabled.
## Understanding Reproducible Output
The extension is designed to generate repeatable data sets. Each function takes `seed` and `nvalues` parameters:
* `seed`: Determines a subset of possible values to generate
* `nvalues`: Determines the number of distinct values
This allows you to generate the same set of values consistently when needed, though the order of values may vary.
## Examples
### String Generation
```sql theme={null}
-- Generate random strings with specified length range
-- This will generate 10 random strings between 5 and 10 characters long
SELECT random_string(
42, -- seed for reproducibility
1000, -- size of pool of possible values
5, -- minimum string length
10 -- maximum string length
) from generate_series(1, 10);
-- If we set a small pool size, we will get less unique values
-- This will return the same two strings 10 times
SELECT random_string(42,2,5,10) from generate_series(1, 10);
```
### Numeric Types
```sql theme={null}
-- Generate 10 random 32-bit integers between 1 and 100
SELECT random_int(42,1000,1,100) from generate_series(1, 10);
-- Generate 10 random double precision numbers between 0 and 1
SELECT random_double_precision(42,1000,0.0,1.0) from generate_series(1, 10);
```
### Network Types
```sql theme={null}
-- Generate 10 random IP addresses
SELECT random_inet(42,1000) from generate_series(1, 10);
-- Generate 10 random CIDR addresses
SELECT random_cidr(42,1000) from generate_series(1, 10);
```
## Common Use Cases
The random extension is invaluable for creating reproducible test datasets, populating development
databases, generating real-looking data for demos and conducting performance testing with controlled data.
## Best Practices
1. **Seed Management**
* Use consistent seeds for reproducible results
* Document seeds used in test scenarios
* Vary seeds systematically for different test cases
2. **Value Distribution**
* Consider the actual number of distinct values needed
* Account for potential PRNG collisions
* Use appropriate ranges for your use case
## Additional Resources
The random extension is a powerful tool for generating reproducible data.
It contains many more functions than what is shown here. To see the full list of functions,
please refer to the [Random Extension Repository](https://github.com/tvondra/random).
# RDKit
Source: https://thenile.dev/docs/extensions/rdkit
RDKit extension for chemical structure handling in PostgreSQL
RDKit is a powerful open-source cheminformatics and machine learning toolkit that provides PostgreSQL
with the ability to handle and analyze chemical structures within the database.
Your Nile database arrives with the RDKit extension already enabled.
## Overview
The RDKit PostgreSQL extension adds support for:
* Chemical structure storage and retrieval
* Substructure and similarity searching
* Chemical structure manipulation
* Molecular descriptor calculation
* Chemical reaction handling
RDKit provides several custom data types:
* `mol` - Represents a molecule (constracted from SMILES notation)
* `qmol` - Represents a molecule containing query information (constructed from SMARTS notation)
* `sfp` - Represents a sparse vector fingerprint
* `bfp` - Represents a bit vectorfingerprint
## Basic Operations
1. **Creating Molecules**
```sql theme={null}
-- Create a molecule from SMILES notation
SELECT mol_from_smiles('CCO') AS ethanol;
-- Create a molecule from SMARTS pattern
SELECT qmol_from_smarts('[OH]') AS hydroxyl;
```
2. **Molecular Properties**
```sql theme={null}
-- Calculate molecular weight
SELECT mol_amw(mol_from_smiles('CCO')) AS molecular_weight;
-- Count atoms
SELECT mol_numatoms(mol_from_smiles('CCO')) AS atom_count;
```
3. **Substructure Searching**
```sql theme={null}
-- Check if a molecule contains a substructure
SELECT mol_from_smiles('CCO') @> qmol_from_smarts('[OH]') AS has_hydroxyl;
```
### Similarity Searching
RDKit supports various similarity metrics:
```sql theme={null}
-- Calculate Tanimoto similarity between molecules
SELECT tanimoto_sml(
morganbv_fp(mol_from_smiles('CCO')),
morganbv_fp(mol_from_smiles('CCN'))
) AS similarity;
```
## Use Cases
RDKit is particularly useful for:
* Drug discovery and development
* Chemical database management
* Structure-activity relationship analysis
* Chemical similarity searching
* Reaction prediction and analysis
## Performance Optimization
For better performance when working with large chemical databases:
1. Create indexes on chemical structure columns:
```sql theme={null}
CREATE INDEX idx_molecule_substructure ON your_table USING gist(molecule);
```
2. Use appropriate fingerprint types for your specific use case:
* Morgan fingerprints for general similarity searching
* MACCS keys for substructure screening
* Topological fingerprints for specific pattern matching
## Additional Resources
* [RDKit Documentation](https://www.rdkit.org/docs/)
* [RDKit PostgreSQL Cartridge](https://www.rdkit.org/docs/Cartridge.html)
* [Chemical Development Kit](https://cdk.github.io/)
# Seg
Source: https://thenile.dev/docs/extensions/seg
Line segment and floating-point interval data type for PostgreSQL
The `seg` extension provides support for representing line segments or floating-point intervals in PostgreSQL.
This type represents both ranges of values and also measurements with uncertainty, randomness or tolerances.
Your Nile database arrives with the seg extension already enabled.
## Overview
The seg extension adds a new data type `seg` that can represent:
* Line segments on a number line
* Floating-point intervals
* Exact or inexact bounds
* Infinite bounds using `< x` or `> x` notation
## Data Type Format
The `seg` data type accepts several input formats:
| Format | Example | Description |
| -------- | ------------ | ------------------------------------------- |
| `x` | `5.0` | Single value, a point |
| `x .. y` | `1.0 .. 2.0` | Interval from x to y |
| `x ..` | `1.0 ..` | Everything greater than x |
| `.. y` | `.. 2.0` | Everything less than y |
| `x` | `>5.0` | A point at X. `>` is preserved as a comment |
| `~x` | `~5.0` | A point at X. `~` is preserved as a comment |
| `x(+-)d` | `5.0(+-)0.1` | Interval from x-d to x+d |
## Examples
### Creating Segments
```sql theme={null}
-- Create basic intervals
SELECT '-1..1'::seg AS basic_interval;
SELECT '..5'::seg AS up_to_five;
SELECT '5..'::seg AS five_and_above;
-- Create approximate points
SELECT '~5.0'::seg AS approximately_five;
```
### Comparison Operations
```sql theme={null}
-- Check if segments overlap
SELECT '-1..1'::seg && '0..2'::seg AS overlaps;
-- Check if segment contains another
SELECT '-1..1'::seg @> '0..0.5'::seg AS contains;
-- Check if segment is contained by another
SELECT '0..0.5'::seg <@ '-1..1'::seg AS is_contained;
-- This returns false because the segment is not contained by the other
SELECT '-1..1'::seg @> '2..3'::seg AS contains;
```
### Working with Measurement Ranges
```sql theme={null}
CREATE TABLE measurements (
id serial PRIMARY KEY,
value seg
);
-- Insert measurements with tolerances
INSERT INTO measurements (value) VALUES
('~10.5'), -- Approximately 10.5
('10.3..10.7'), -- Between 10.3 and 10.7
('..11.0'), -- Less than 11.0
('9.5..'); -- Greater than 9.5
-- Find overlapping measurements
SELECT a.value, b.value
FROM measurements a, measurements b
WHERE a.value && b.value and a.value < b.value;
```
## Use Cases
The seg extension can be used with any type of data that can be represented as a floating-point interval.
But it is most useful for recording laboratory measurements with uncertainty or tolerances.
## Performance Optimization
For better query performance with seg data:
1. Create GiST indexes on seg columns:
```sql theme={null}
-- Create a table with a seg column
CREATE TABLE temperature_readings (
id serial PRIMARY KEY,
location text,
temp_range seg
);
-- Create a GiST index on the seg column
CREATE INDEX idx_temp_range ON temperature_readings USING gist(temp_range);
```
2. Common operators that can use the GiST index:
* `<<` (strictly left of)
* `>>` (strictly right of)
* `&<` (does not extend right of)
* `&>` (does not extend left of)
* `&&` (overlaps)
* `@>` (contains)
* `<@` (contained in)
## Additional Resources
* [PostgreSQL seg Documentation](https://www.postgresql.org/docs/current/seg.html)
* [GiST Index Documentation](https://www.postgresql.org/docs/current/gist.html)
# Tablefunc
Source: https://thenile.dev/docs/extensions/tablefunc
Table function utilities for cross tabulation and pivot operations
The `tablefunc` extension provides a set of functions for manipulating tables, including cross tabulation, pivoting, and connecting tables.
Your Nile database arrives with the tablefunc extension already enabled.
## Overview
The tablefunc extension provides several useful functions:
* `crosstab`: Creates pivot tables and cross tabulations
* `normal_rand`: Generates normally distributed random numbers
* `connectby`: Implements hierarchical queries
## Cross Tabulation Functions
### Basic Crosstab
The `crosstab` function transforms row-oriented data into a cross-tabulation format (pivot table).
```sql theme={null}
-- Basic crosstab syntax
SELECT * FROM crosstab(
source_sql text, -- SQL query returning (row_name, category, value)
category_sql text -- SQL query returning distinct categories
) AS ct (
row_name text, -- Name of the row
category1 text, -- First category column
category2 text, -- Second category column
...
);
```
### Example: Sales by Quarter
```sql theme={null}
-- Create sample data
CREATE TABLE quarterly_sales (
tenant_id uuid,
year int,
quarter text,
sales numeric
);
INSERT INTO quarterly_sales VALUES
('11111111-1111-1111-1111-111111111111', 2023, 'Q1', 100),
('11111111-1111-1111-1111-111111111111', 2023, 'Q2', 150),
('11111111-1111-1111-1111-111111111111', 2023, 'Q3', 130),
('11111111-1111-1111-1111-111111111111', 2023, 'Q4', 180),
('11111111-1111-1111-1111-111111111111', 2024, 'Q1', 120),
('11111111-1111-1111-1111-111111111111', 2024, 'Q2', 160);
-- Create cross tab of sales by year and quarter
SELECT * FROM crosstab(
'SELECT year, quarter, sales
FROM quarterly_sales
ORDER BY 1,2',
'SELECT DISTINCT quarter
FROM quarterly_sales
ORDER BY 1'
) AS ct (
year int,
"Q1" numeric,
"Q2" numeric,
"Q3" numeric,
"Q4" numeric
);
```
Result:
```
year | Q1 | Q2 | Q3 | Q4
------+-----+-----+-----+-----
2023 | 100 | 150 | 130 | 180
2024 | 120 | 160 | null| null
```
## Normal Random Numbers
The `normal_rand` function generates normally distributed random numbers:
```sql theme={null}
-- Generate 5 normal random numbers
SELECT * FROM normal_rand(
5, -- number of rows
15 -- standard deviation
);
```
## Hierarchical Queries
The `connectby` function helps create hierarchical queries.
Let's create a sample employee hierarchy and query it:
```sql theme={null}
-- Create sample employee hierarchy
CREATE TABLE employees (
tenant_id uuid,
employee_id int,
name text,
manager_id int,
PRIMARY KEY (tenant_id, employee_id)
);
INSERT INTO employees VALUES
('11111111-1111-1111-1111-111111111111', 1, 'CEO', NULL),
('11111111-1111-1111-1111-111111111111', 2, 'VP Sales', 1),
('11111111-1111-1111-1111-111111111111', 3, 'VP Engineering', 1),
('11111111-1111-1111-1111-111111111111', 4, 'Sales Manager', 2),
('11111111-1111-1111-1111-111111111111', 5, 'Engineer', 3);
-- Query hierarchical employee structure
-- Order of results is not guaranteed
SELECT * FROM connectby(
'employees', 'employee_id', 'manager_id', '1', 0, '>')
AS t(employee_id int, manager_id int, level int, branch text);
```
Result:
```
employee_id | manager_id | level | branch
-------------+------------+-------+--------
1 | | 0 | 1
2 | 1 | 1 | 1>2
4 | 2 | 2 | 1>2>4
3 | 1 | 1 | 1>3
5 | 3 | 2 | 1>3>5
```
## Additional Resources
Tablefunc is a powerful extension for working with tables and hierarchies.
We only covered a few basic examples here, but there are many more options available.
You can read the rest in [PostgreSQL tablefunc Documentation](https://www.postgresql.org/docs/current/tablefunc.html)
# Unit
Source: https://thenile.dev/docs/extensions/unit
SI unit conversion and dimensional analysis for PostgreSQL
The `unit` extension adds support for SI (International System of Units) measurements and conversions in PostgreSQL.
Your Nile database arrives with the unit extension already enabled.
## Overview
The unit extension provides:
* Storage and manipulation of SI units and measurements
* Automatic unit conversion
* Dimensional analysis
* Support for SI prefixes (kilo, milli, etc.)
* Mathematical operations with units
## Basic Usage
### Creating Unit Values
The `unit` type accepts values in the format `number unit`:
```sql theme={null}
-- Basic unit values
SELECT '100 m'::unit AS length; -- 100 meters
SELECT '50 kg'::unit AS mass; -- 50 kilograms
SELECT '10 m/s'::unit AS velocity; -- 10 meters per second
SELECT '9.81 m/s^2'::unit AS gravity; -- 9.81 meters per second squared
```
### Unit Conversions
Units can be converted using the `@>` operator or the `unit_transform` function:
```sql theme={null}
-- Convert meters to kilometers
SELECT '1000 m'::unit @ 'km' AS kilometers; -- Returns: 1 km
-- Convert kilometers to meters
SELECT '1 km'::unit @ 'm' AS meters; -- Returns: 1000 m
```
## Mathematical Operations
The unit extension supports basic mathematical operations while maintaining dimensional correctness:
```sql theme={null}
-- Addition and subtraction (must have same dimensions)
SELECT '1 km'::unit + '100 m'::unit AS total_distance; -- Returns: 1.1 km
SELECT '1 hour'::unit - '30 min'::unit AS time_diff; -- Returns: 30 min
-- Multiplication
SELECT '10 m'::unit * '5 m'::unit AS area; -- Returns: 50 m²
SELECT '50 kg'::unit * '9.81 m/s^2'::unit AS force; -- Returns: 490.5 kg⋅m/s²
-- Division
SELECT '100 m'::unit / '10 s'::unit AS speed; -- Returns: 10 m/s
SELECT '500 m'::unit / '2 m'::unit AS ratio; -- Returns: 250 (dimensionless)
```
## Working with Measurements in Tables
```sql theme={null}
-- Create a table with unit columns
CREATE TABLE physical_measurements (
tenant_id uuid,
id int,
distance unit,
mass unit,
temperature unit,
PRIMARY KEY (tenant_id, id)
);
-- Insert measurements
INSERT INTO physical_measurements (tenant_id, id, distance, mass, temperature) VALUES
('11111111-1111-1111-1111-111111111111', 1, '100 m', '75 kg', '37 degC'),
('11111111-1111-1111-1111-111111111111', 2, '2.5 km', '80 kg', '36.5 degC');
-- Query with conversions
SELECT
distance @ 'm' AS distance_meters,
mass @ 'g' AS mass_grams,
temperature @ 'degF' AS temp_fahrenheit
FROM physical_measurements;
```
## Error Handling
The unit extension enforces dimensional correctness:
```sql theme={null}
-- These will raise errors
SELECT '1 m'::unit + '1 kg'::unit; -- Error: dimension mismatch
SELECT '100 m^2'::unit @ 'm'; -- Error: dimension mismatch
SELECT '10 m/s'::unit * '5 kg/m'::unit; -- Works: results in 50 kg/s
```
## Additional Resources
Unit extension supports a large number of units and conversions,
and we've only scratched the surface of what it can do.
You can find more information in the [unit extension repository](https://github.com/df7cb/postgresql-unit).
The extension uses (among others) the unit definitions from Gnu Units.
The [unit definition file](https://github.com/df7cb/postgresql-unit/blob/master/definitions.units) has comprehensive documentation on the 2400 units it supports.
# UUID-OSSP
Source: https://thenile.dev/docs/extensions/uuid-ossp
UUID generation functions for PostgreSQL
The `uuid-ossp` extension provides functions for generating Universally Unique Identifiers (UUIDs) in PostgreSQL.
Your Nile database arrives with the uuid-ossp extension already enabled.
## Overview
The uuid-ossp extension provides several functions for generating UUIDs according to different standards:
* Version 1: Time-based UUIDs
* Version 3: Namespace and name-based UUIDs using MD5
* Version 4: Random UUIDs
* Version 5: Namespace and name-based UUIDs using SHA-1
Nile includes `public.uuid_generate_v7()` which generates UUIDs with
time-ordered lexicographically sortable strings. It is recommended to use this
function for fields that are used in sorting and indexing.
## UUID Generation Functions
### UUID Version 4 (Random)
The most commonly used function is `uuid_generate_v4()`, which generates a random UUID:
```sql theme={null}
-- Generate a random UUID
SELECT uuid_generate_v4();
-- Result: 123e4567-e89b-12d3-a456-426614174000 (example)
-- Use in a table
CREATE TABLE documents (
tenant_id uuid,
document_id uuid DEFAULT uuid_generate_v4(),
title text,
content text,
PRIMARY KEY (tenant_id, document_id)
);
-- Insert with auto-generated UUID
INSERT INTO documents (tenant_id, title, content) VALUES
('11111111-1111-1111-1111-111111111111', 'My Document', 'Content here') returning document_id;
```
### UUID Version 1 (Time-based)
`uuid_generate_v1()` creates a UUID based on the current timestamp and MAC address:
```sql theme={null}
-- Generate a time-based UUID
SELECT uuid_generate_v1();
-- Also available: uuid_generate_v1mc()
-- Similar to v1 but uses a random multicast MAC address
SELECT uuid_generate_v1mc();
```
### UUID Version 3 (Name-based, MD5)
`uuid_generate_v3()` creates a UUID based on a namespace and name using MD5:
```sql theme={null}
-- Generate a UUID from namespace and name using MD5
SELECT uuid_generate_v3(
'a0eebc99-9c0b-1ef8-b1ff-826046d7f000'::uuid, -- namespace
'example.com' -- name
);
-- Common namespace UUIDs
SELECT uuid_generate_v3(uuid_ns_dns(), 'example.com'); -- DNS namespace
SELECT uuid_generate_v3(uuid_ns_url(), 'http://example.com'); -- URL namespace
SELECT uuid_generate_v3(uuid_ns_oid(), '1.2.3.4'); -- OID namespace
SELECT uuid_generate_v3(uuid_ns_x500(), 'CN=Example'); -- X500 namespace
```
### UUID Version 5 (Name-based, SHA-1)
`uuid_generate_v5()` creates a UUID based on a namespace and name using SHA-1:
```sql theme={null}
-- Generate a UUID from namespace and name using SHA-1
SELECT uuid_generate_v5(
'a0eebc99-9c0b-1ef8-b1ff-826046d7f000'::uuid, -- namespace
'example.com' -- name
);
-- Common namespace UUIDs
SELECT uuid_generate_v5(uuid_ns_dns(), 'example.com'); -- DNS namespace
SELECT uuid_generate_v5(uuid_ns_url(), 'http://example.com'); -- URL namespace
SELECT uuid_generate_v5(uuid_ns_oid(), '1.2.3.4'); -- OID namespace
SELECT uuid_generate_v5(uuid_ns_x500(), 'CN=Example'); -- X500 namespace
```
## Common Use Cases
### Primary Keys
```sql theme={null}
-- Using UUID as primary key
CREATE TABLE contacts (
tenant_id uuid,
contact_id uuid DEFAULT uuid_generate_v4(),
name text,
email text,
PRIMARY KEY (tenant_id, contact_id)
);
-- Insert with auto-generated UUID
INSERT INTO contacts (tenant_id, name, email) VALUES
('11111111-1111-1111-1111-111111111111', 'john_doe', 'john@example.com') returning contact_id;
```
### Deterministic IDs
```sql theme={null}
-- Generate consistent UUIDs for the same input
CREATE TABLE products (
tenant_id uuid,
product_id uuid,
sku text,
name text,
PRIMARY KEY (tenant_id, product_id)
);
-- Insert with deterministic UUID based on SKU
-- If you run this multiple times, you'll get duplicate key error
INSERT INTO products (tenant_id, product_id, sku, name) VALUES
('11111111-1111-1111-1111-111111111111',
uuid_generate_v5(uuid_ns_url(), 'SKU123'),
'SKU123',
'Product Name') returning product_id;
```
## Additional Resources
* [PostgreSQL UUID Documentation](https://www.postgresql.org/docs/current/uuid-ossp.html)
* [UUID RFC 9562](https://www.rfc-editor.org/rfc/rfc9562.html)
# Pgvector
Source: https://thenile.dev/docs/extensions/vector
Vector extension for PostgreSQL
The **`pgvector`** extension in PostgreSQL is used to efficiently store and query vector data. The **`pgvector`** extension provides
PostgreSQL with the ability to store and perform operations on vectors directly within the database.
Nile supports **`pgvector`** out of the box on the latest version - `0.8.0`.
Pgvector lets you store and query vectors directly within your usual Postgres database - with the rest of your data. This is both convenient and efficient. It supports:
* Exact and approximate nearest neighbor search (with optional HNSW and IVFFlat indexes)
* Single-precision, half-precision, binary, and sparse vectors
* L2 distance, inner product, cosine distance, L1 distance, Hamming distance, and Jaccard distance
* Any language with a Postgres client
Plus ACID compliance, point-in-time recovery, JOINs, and all of the other great features of Postgres
## Create tenant table with vector type
Vector types work like any other standard types. You can make them the type of a column in a tenant table and Nile will take care of isolating the embeddings per tenant.
```sql theme={null}
-- creating a table to store wiki documents for a Notion like
-- SaaS application with vector dimension of 3
CREATE TABLE wiki_documents(
tenant_id uuid,
id integer,
embedding vector(3)
);
```
## Store vectors per tenant
Once you have the table defined, you would want to populate the embeddings. Typically, this is done by querying a large language model (eg. OpenAI, HuggingFace), retrieving the embeddings and storing them in the vector store. Once stored, the embeddings follow the standard tenant rules. They can be isolated, sharded and placed based on the tenant they belong to.
```sql theme={null}
INSERT INTO wiki_documents (tenant_id,id, embedding)
VALUES ('018ade1a-7843-7e60-9686-714bab650998',1, '[1,2,3]');
```
## Query vectors
Pgvector supports 6 types of vector similarity operators:
Operator
Name
Description
Use Cases
\<->
vector\_l2\_ops
L2 distance. Measure of the straight-line distance between two points in
a multi-dimensional space. It calculates the length of the shortest path
between the points, which corresponds to the hypotenuse of a right
triangle.
Used in clustering, k-means clustering, and distance-based
classification algorithms
\<#>
vector\_ip\_ops
Inner product. The inner product, also known as the dot product,
measures the similarity or alignment between two vectors. It calculates
the sum of the products of corresponding elements in the vectors.
Used in similarity comparison or feature selection. Note that for
normalized vectors, inner product will result in the same ranking as
cosine distance, but is more efficient to calculate. So this is a good
choice if you use an embedding algorith that produces normalized vectors
(such as OpenAI's)
\<=>
vector\_cosine\_ops
Cosine distance. Cosine distance, often used as cosine similarity when
measuring similarity, quantifies the cosine of the angle between two
vectors in a multi-dimensional space. It focuses on the direction rather
than the magnitude of the vectors.
Used in text similarity, recommendation systems, and any context where
you want to compare the direction of vectors
\<+>
vector\_l1\_ops
L1 distance. The L1 distance, also known as the Manhattan distance,
measures the distance between two points in a grid-like path (like a
city block). It is the distance between two points measured along axes
at right angles.
Less sensitive to outliers than L2 distance and according to some
research, better for high-dimensional data.
\<\~>
bit\_hamming\_ops
Hamming distance. The Hamming distance measures the number of positions
at which the corresponding symbols are different.
Used with binary vectors. Mostly for discrete data like categories. Also
used for error-correcting codes and data compression.
\<%>
bit\_jaccard\_ops
Jaccard distance. Measures similarity between sets by calculating the
ratio of the intersection to the union of the two sets (how many
positions are the same out of the total positions).
Used with binary vectors. Useful for comparing customers purchase
history, recommendation systems, similarites in terms used in different
texts, etc.
You could use any one of them to find the distance between vectors. The choice of operator depends not only on the use-case, but also on
the model used to generate the embeddings. For example, OpenAI's models return normalized embeddings, so using inner product or cosine distance
will give the same results and inner product is more efficient. However, some models return non-normalized embeddings, so cosine distance should be used.
Real world vectors are quite large - 768 to 4096 dimensions are not uncommon. But for the sake of the example, we'll use small vectors:
To get the L2 distance
```sql theme={null}
SELECT embedding <-> '[3,1,2]' AS distance FROM wiki_documents;
```
For inner product, multiply by -1 (since `<#>` returns the negative inner product)
```sql theme={null}
SELECT (embedding <#> '[3,1,2]') * -1 AS inner_product FROM wiki_documents;
```
For cosine similarity, use 1 - cosine distance
```sql theme={null}
SELECT 1 - (embedding <=> '[3,1,2]') AS cosine_similarity FROM wiki_documents;
```
## Vector Indexes
`pgvector` supports two types of indexes:
* HNSW
* IVFFlat
Keep in mind that vector indexes are unlike other database indexes. They are used to perform efficient **approximate** nearest neighbor searches.
Without vector indexes (also called **flat indexes** by other vector stores), queries sequentially scan through all vectors for the given query,
and compute the distance to the query vector. This is computationally expensive, but guarantees to find the nearest neighbors
(also called **exact nearest neighbors** or **KNN**).
With vector indexes, the query will search a subset of the vectors that is expected to contain the nearest neighbors (but may not contain all of them).
This is computationally efficient, but the results are not guaranteed to be the nearest neighbors.
When using vector indexes, you can control the trade-off between speed and recall by specifying the index type and parameters.
### HNSW
An HNSW index creates a multilayer graph. It has slower build times and uses more memory than IVFFlat, but has better query performance
(in terms of speed-recall tradeoff). There’s no training step like IVFFlat, so the index can be created without any data in the table.
When creating HNSW, you can specify the maximum number of connections in a layer (`m`) and the number of candidate vectors considered
when building the graph (`ef_construction`). More connections and more candidate vectors will improve recall but will increase build time and memory.
If you don't specify the parameters, the default values are `m = 16` and `ef_construction = 64`.
Add an index for each distance function you want to use.
```sql theme={null}
CREATE INDEX ON wiki_documents USING hnsw (embedding vector_l2_ops);
```
Inner product
```sql theme={null}
CREATE INDEX ON wiki_documents USING hnsw (embedding vector_ip_ops);
```
Cosine distance
```sql theme={null}
CREATE INDEX ON wiki_documents USING hnsw (embedding vector_cosine_ops);
```
Specifying the parameters
```sql theme={null}
CREATE INDEX ON wiki_documents USING hnsw (embedding vector_cosine_ops) WITH (m = 16, ef_construction = 100);
```
While querying, you can specify the size of the candidate list that will be searched (`hnsw_ef`):
```sql theme={null}
SET hnsw_ef = 100;
SELECT * FROM wiki_documents ORDER BY embedding <=> '[3,1,2]' LIMIT 10;
```
Vectors with up to 2,000 dimensions can be indexed.
### IVFLAT
An IVFFlat index divides vectors into lists, and then searches a subset of those lists that are closest to the query vector.
It has faster build times and uses less memory than HNSW, but has lower query performance (in terms of speed-recall tradeoff).
Three keys to achieving good recall are:
1. Create the index **after** the table has some data
2. Choose an appropriate number of lists - a good place to start is `rows / 1000` for up to 1M rows and `sqrt(rows)` for over 1M rows.
3. When querying, specify an appropriate number of probes (higher is better for recall, lower is better for speed) - a good place to start is `sqrt(lists)`
Add an index for each distance function you want to use.
L2 distance
```sql theme={null}
CREATE INDEX ON wiki_documents USING ivfflat (embedding vector_l2_ops) WITH (lists = 100);
```
Inner product
```sql theme={null}
CREATE INDEX ON wiki_documents USING ivfflat (embedding vector_ip_ops) WITH (lists = 100);
```
Cosine distance
```sql theme={null}
CREATE INDEX ON wiki_documents USING ivfflat (embedding vector_cosine_ops) WITH (lists = 100);
```
When querying, you can specify the number of probes (i.e. how many lists to search):
```sql theme={null}
SET ivfflat_probes = 100;
SELECT * FROM wiki_documents ORDER BY embedding <=> '[3,1,2]' LIMIT 10;
```
More probes will improve recall but will slow down the query.
Vectors with up to 2,000 dimensions can be indexed.
## Filtering
Typically, vector search is used to find the nearest neighbors, which means that you would limit the number after ordering by distance:
```sql theme={null}
SELECT * FROM wiki_documents ORDER BY embedding <=> '[3,1,2]' LIMIT 10;
```
It is a good idea to also filter by distance, so you won't include vectors that are too far away, even if they are the nearest neighbors.
This will prevent you from getting nonsensical results in cases where there isn't much data in the vector store.
```sql theme={null}
SELECT * FROM wiki_documents
WHERE embedding <=> '[3,1,2]' < 0.9
ORDER BY embedding <=> '[3,1,2]'
LIMIT 10;
```
Nile will automatically limit the results to only the vectors that belong to the current tenant:
```sql theme={null}
SET nile.tenant_id = '018ade1a-7843-7e60-9686-714bab650998';
SELECT * FROM wiki_documents
WHERE embedding <=> '[3,1,2]' < 0.9
ORDER BY embedding <=> '[3,1,2]'
LIMIT 10;
```
And you can also filter results by additional criteria (some vector stores call this "metadata filtering" or "post-filtering"):
```sql theme={null}
SELECT * FROM wiki_documents
WHERE embedding <=> '[3,1,2]' < 0.9
AND category = 'product'
ORDER BY embedding <=> '[3,1,2]'
LIMIT 10;
```
It is recommended to add indexes on the columns used for filtering, so the query can be optimized.
Especially when the use of the filter can lead to small enough result sets that a sequential scan is faster than use of the vector index.
This will optimize not just performance but also recall.
With approximate indexes, filtering is applied after the index is scanned. If a condition matches 10% of rows,
with HNSW and the default hnsw\.ef\_search of 40, only 4 rows will match on average.
Starting in version `0.8.0`, you can enable **iterative index scans**, which will automatically scan more of the index when needed.
### Iterative Index scans
Using iterative index scans, Postgres will scan the approximate index for nearest neighbors, apply additional filters and, if the number of
neighbors after filtering is insufficient, it will continue scanning until sufficient results are found.
Each index has its own configuration (GUC) for iterative scans: `hnsw.iterative_scan` and `ivfflat.iterative_scan`.
By default both configurations are set to `off`.
HNSW indexes support both relaxed and strict ordering for the iterative scans. Strict order guarantees that the returned results are ordered by exact distance.
Relaxed order allows results that are slightly out of order, but provides better recall (i.e. fewer missed results due to the approximate
nature of the index).
```sql theme={null}
SET hnsw.iterative_scan = strict_order;
-- or
SET hnsw.iterative_scan = relaxed_order;
```
IVFFlat indexes only allow relaxed ordering:
```sql theme={null}
SET ivfflat.iterative_scan = relaxed_order;
```
Once you set these configs, you don't need to change your existing queries. You should immediately see the same queries return the correct number of results.
However, if you use relaxed ordering, you can re-order the result using materialized CTE:
```sql theme={null}
WITH relaxed_results AS MATERIALIZED (
SELECT id, embedding <-> '[1,2,3]' AS distance FROM items WHERE category_id = 123 ORDER BY distance LIMIT 5
) SELECT * FROM relaxed_results ORDER BY distance;
```
If you filter by distance (recommended, to avoid nonsense results in case there aren't many similar vectors), it is recommended to use
materialized CTE and place the filter outside the CTE in order to avoid overscanning:
```sql theme={null}
WITH nearest_results AS MATERIALIZED (
SELECT id, embedding <=> '[1,2,3]' AS distance FROM items ORDER BY distance LIMIT 5
) SELECT * FROM nearest_results WHERE distance < 1 ORDER BY distance;
```
Even with iterative scans, pgvector limits the index scan in order to balance time, resource use and recall.
Increasing these limits can increase query latency but potentially improve recall:
HSNW has a configuration that controls the total number of rows that will be scanned by a query across all iterations (20,000 is the default):
```sql theme={null}
SET hnsw.max_scan_tuples = 20000;
```
IVFFLat lets you configure the maximum number of lists that will be checked for nearest neighbors:
```sql theme={null}
SET ivfflat.max_probes = 100;
```
## Quantization
*Introduced in pgvector 0.7.0*
Quantization is a technique of optimizing vector storage and query performance by using fewer bits to store the vectors.
By default, pgvector's Vector type is in 32-bit floating point format. The `halfvec` data type uses 16-bit floating point format, which has the following benefits:
* Reduced storage requirements (half the memory)
* Faster query performance
* Reduced index size (both in disk and memory)
* Can index vectors with up to 4096 dimensions (which covers the most popular embedding models)
To use `halfvec`, you can create a table with the `halfvec` type:
```sql theme={null}
CREATE TABLE wiki_documents(
tenant_id uuid,
id integer,
embedding halfvec(3) -- put the real number of dimensions here
);
```
You can also create quantized indexes on regular sized vectors:
```sql theme={null}
CREATE INDEX ON wiki_documents USING hnsw ((embedding::halfvec(3)) halfvec_l2_ops);
```
In this case, you need to cast the vector to `halfvec` in the query:
```sql theme={null}
SELECT * FROM wiki_documents WHERE embedding::halfvec(3) <=> '[3,1,2]' LIMIT 10;
```
Note that to create an index on `halfvec`, you need to specify the distance function as `halfvec_l2_ops` or `halfvec_cosine_ops`.
## Sparse Vectors
*Introduced in pgvector 0.7.0*
Sparse vectors are vectors in which the values are mostly zero. These are common in text search algorithms, where each dimension represents
a word and the value represents the relative frequency of the word in the document - BM25, for example. Some embedding models, such as BGE-M3, also use sparse vectors.
Pgvector supports sparse vector type `sparsevec` and the associated similarity operators.
Because sparse vectors can be extremely large but most of the values are zero, pgvector stores them in a compressed format.
```sql theme={null}
CREATE TABLE wiki_documents(
tenant_id uuid,
id integer,
embedding sparsevec(5) -- put the real number of dimensions here
);
INSERT INTO wiki_documents (tenant_id, id, embedding)
VALUES ('018ade1a-7843-7e60-9686-714bab650998', 1, '{1:1,3:2,5:3}/5');
SELECT * FROM wiki_documents ORDER BY embedding <-> '{1:3,3:1,5:2}/5' LIMIT 5;
```
The format is `{index1:value1,index2:value2,...}/N`, where N is the number of dimensions and the indices start from 1 (like SQL arrays).
Because the format is a bit unusual, it is recommended to use [pgvector's libraries for your favorite language](https://github.com/pgvector/pgvector?tab=readme-ov-file#languages)
to insert and query sparse vectors.
## Summary
You can read more about pgvector on their [github](https://github.com/pgvector/pgvector/blob/master/README.md)
If you have any feedback or questions on building AI-native SaaS applications on Nile, please do reach out on our [Github discussion forum](https://github.com/orgs/niledatabase/discussions) or our [Discord community](https://discord.gg/8UuBB84tTy).
# Xicor
Source: https://thenile.dev/docs/extensions/xicor
Incremental correlation calculations in PostgreSQL
The `xicor` extension provides support for calculating the Xi correlation coefficient, a robust measure of correlation that works well with non-linear relationships.
Your Nile database arrives with the xicor extension already enabled.
## Overview
The xicor extension provides:
* Calculation of Xi correlation coefficient
* Support for incremental correlation updates
* Robust handling of non-linear relationships
* Better detection of dependencies between variables compared to Pearson correlation
## Basic Usage
Calculate correlations:
```sql theme={null}
-- Create a table with grouped measurements
CREATE TABLE group_measurements (
tenant_id uuid,
group_id int,
x numeric,
y numeric,
PRIMARY KEY (tenant_id, group_id, x)
);
-- Insert grouped data
INSERT INTO group_measurements (tenant_id, group_id, x, y) VALUES
('11111111-1111-1111-1111-111111111111', 1, 1, 1),
('11111111-1111-1111-1111-111111111111', 1, 2, 4),
('11111111-1111-1111-1111-111111111111', 1, 3, 9),
('11111111-1111-1111-1111-111111111111', 2, 1, 2),
('11111111-1111-1111-1111-111111111111', 2, 2, 3),
('11111111-1111-1111-1111-111111111111', 2, 3, 5);
-- Calculate Xi correlation
SELECT xicor(x, y) FROM measurements;
-- Calculate correlation by group
SELECT
group_id,
xicor(x, y) as correlation
FROM group_measurements
GROUP BY group_id;
```
## Understanding Xi Correlation
The Xi correlation coefficient has several advantages over traditional correlation measures:
1. Robust to Non-linearity:
```sql theme={null}
-- Xi correlation detects monotonic relationships
-- even when they're not linear
SELECT xicor(x, exp(x)) FROM generate_series(1, 5) as x;
```
2. Range of Values:
* Returns values between 0 and 1
* 0 indicates no correlation
* 1 indicates perfect correlation (monotonic relationship)
3. Interpretation:
```sql theme={null}
-- Perfect correlation (monotonic)
SELECT xicor(x, x) FROM generate_series(1, 5) as x; -- Returns 1.0
-- No correlation (random)
SELECT xicor(x, random()) FROM generate_series(1, 1000) as x; -- Returns ~0
```
## Use Cases
### Financial Analysis
```sql theme={null}
-- Analyze stock price correlations
CREATE TABLE stock_prices (
tenant_id uuid,
date date,
stock_symbol text,
price numeric,
PRIMARY KEY (tenant_id, date, stock_symbol)
);
-- Calculate correlation between stock prices
SELECT
s1.stock_symbol as stock1,
s2.stock_symbol as stock2,
xicor(s1.price, s2.price) as price_correlation
FROM stock_prices s1
JOIN stock_prices s2 ON s1.date = s2.date
WHERE s1.stock_symbol < s2.stock_symbol
GROUP BY s1.stock_symbol, s2.stock_symbol;
```
### Scientific Measurements
```sql theme={null}
-- Analyze sensor data correlations
CREATE TABLE sensor_readings (
tenant_id uuid,
timestamp timestamp,
sensor_id text,
temperature numeric,
humidity numeric,
PRIMARY KEY (tenant_id, timestamp, sensor_id)
);
-- Calculate correlation between temperature and humidity
SELECT
sensor_id,
xicor(temperature, humidity) as temp_humidity_correlation
FROM sensor_readings
GROUP BY sensor_id;
```
## Additional Resources
* [Xi Correlation Extension repository](https://github.com/Florents-Tselai/pgxicor)
* [Xi Correlation Paper](https://arxiv.org/abs/1909.10140)
* [Statistical Correlation Measures in PostgreSQL](https://www.postgresql.org/docs/current/functions-aggregate.html)
# Architecture of Nile Postgres
Source: https://thenile.dev/docs/getting-started/architecture
Nile has built Postgres from the ground up with tenants/customers as a core building block. The key highlights of our design are:
**Decoupled storage and compute**. The compute layer is essentially Postgres, modified to store each tenant's data in separate pages. The storage layer consists of a fleet of machines that house these pages. An external machine stores the log, while both the log and pages are archived in S3 for long-term storage.
**Tenant-aware Postgres pages**. A typical Postgres database comprises objects like tables and indexes, represented by 8KB pages. In Nile, tables are either tenant-specific or shared. Each page of a tenant table belongs exclusively to one tenant, with all records within a page associated with that tenant. This decoupled storage and tenant-dedicated page system allows for instantaneous tenant migration between different Postgres compute instances. Moving a tenant simply involves transferring tenant leadership from one compute instance to another while maintaining references to the same pages in the storage layer.
**Support for both serverless and provisioned compute.** The Postgres compute layer offers two types of compute. Serverless compute is built with true multitenancy, while provisioned compute is dedicated to a single Nile customer. Nile users can have any number of provisioned compute instances in the same database. Tenants can be placed on either serverless or provisioned compute.
**Distributed querying across tenants and a central schema store.** The distributed query layer operates across tenants and functions between serverless and provisioned compute instances. A central schema store employs distributed transactions to apply schemas to every tenant during DDL execution. This ensures correct schema application and enables schema recovery for tenants during failures.
**A global gateway for tenant routing, inter-region communication, and connection pooling.** The gateway uses the Postgres protocol to route requests to different tenants. It can communicate with gateways in other regions and serves as a connection pooling layer, eliminating the need for a separate pooler.
# KnowledgeAI - PDF search assistant for your organization
Source: https://thenile.dev/docs/getting-started/examples/chat_with_pdf
In this tutorial, you will learn how to use Nile's tenant virtualization, user management and
vector embedding features to build a SaaS application that allows users to search the PDF documents
in an organization. The SaaS will allow users to upload a PDF document, and then ask questions about the document.
Using embeddings generated by OpebAI and stored in Nile, and a similarity search using pg\_vector,
to provide GPT-3.5 context that will help answer the questions. The embeddings are stored per tenant and data and
workload is isolated to a tenant.
## 1. Create a database
1. Sign up for an invite to [Nile](https://thenile.dev) if you don't have one already
2. You should see a welcome message. Click on "Lets get started"
3. Give your workspace and database names, or you can accept the default auto-generated names.
## 2. Create a table with pg\_vector extension
Once you created your database, you'll land in Nile's web-based SQL editor. Great place to create the
tables we need for this app. Lets start with the embeddings table.
```sql theme={null}
CREATE TABLE "file_embedding" (
"id" UUID DEFAULT (gen_random_uuid()),
"tenant_id" UUID NOT NULL,
"file_id" UUID NOT NULL,
"embedding_api_id" UUID NOT NULL,
"embedding" vector(1024),
"createdAt" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
"updatedAt" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
"pageContent" TEXT,
"location" TEXT,
CONSTRAINT "file_embedding_pkey" PRIMARY KEY ("id", "tenant_id"),
CONSTRAINT "file_embedding_file_id_fkey" FOREIGN KEY ("file_id", "tenant_id") REFERENCES "file" ("id", "tenant_id")
);
```
This is a tenant-aware table that stores the embeddings of the PDF documents.
You can see that each row belongs to a specific tenant, and that the `embedding`
column is of type `vector(1024)`. Vector type is provided by the pg\_vector extension for storing embeddings.
By storing embeddings in a tenant-aware table, we can use Nile's built-in tenant isolation to ensure that
information about PDFs won't leak between tenants.
## 3. Create metadata tables
We'll need few more tables to store information about the PDF documents, the conversations with them
and the users. Go ahead and create these.
```bash theme={null}
CREATE TABLE "file" (
"id" UUID DEFAULT (gen_random_uuid()),
"tenant_id" UUID NOT NULL,
"url" TEXT,
"key" TEXT,
"createdAt" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
"updatedAt" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
"user_id" UUID NOT NULL,
"user_picture" TEXT,
"user_name" TEXT,
"isIndex" Boolean,
"name" TEXT,
"pageAmt" INTEGER,
CONSTRAINT "file_pkey" PRIMARY KEY ("id", "tenant_id"),
CONSTRAINT "unique_key_per_tenant" UNIQUE ("tenant_id", "key")
);
CREATE TABLE "message" (
"id" UUID DEFAULT (gen_random_uuid()),
"tenant_id" UUID NOT NULL,
"text" TEXT,
"isUserMessage" BOOLEAN,
"createdAt" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
"updatedAt" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
"user_id" UUID NOT NULL,
"user_picture" TEXT,
"user_name" TEXT,
"fileId" UUID,
CONSTRAINT "message_pkey" PRIMARY KEY ("id", "tenant_id"),
CONSTRAINT "message_fileId_fkey" FOREIGN KEY ("fileId", "tenant_id") REFERENCES "file" ("id", "tenant_id")
);
CREATE TABLE "user_subscription" (
"id" UUID DEFAULT (gen_random_uuid()),
"user_id" UUID NOT NULL,
"tenant_id" UUID NOT NULL,
"stripe_customer_id" TEXT,
"stripe_subscription_id" TEXT,
"stripe_price_id" TEXT,
"stripe_current_period_end" TIMESTAMP,
CONSTRAINT "subscription_pkey" PRIMARY KEY ("id", "tenant_id"),
CONSTRAINT "user_subscription_user_id_fkey" FOREIGN KEY ("user_id", "tenant_id") REFERENCES users.tenant_users ("user_id", "tenant_id"),
CONSTRAINT "unique_stripe_customer_id" UNIQUE ("stripe_customer_id", "tenant_id"),
CONSTRAINT "unique_stripe_subscription_id" UNIQUE ("stripe_subscription_id", "tenant_id")
);
```
If all went well, you'll see the new tables in the panel on the left hand side of the query editor. You can also see Nile's built-in tenant table next to it.
### 3. Getting credentials
In the left-hand menu, click on "Settings" and then select "Credentials".
Generate credentials and keep them somewhere safe. These give you access to the database.
### 4. Setting up Google Authentication
This demo uses Google authentication for signup. You will need to configure this in both Google and Nile,
following the instructions [in the example](https://github.com/niledatabase/niledatabase/blob/main/examples/user_management/social_login_google/NextJS/README.md).
### 5. Setting up 3rd Party SaaS
This example requires a few more 3rd party SaaS accounts.
You'll need to set them up and grab API keys to configure this example:
* [UploadThing](https://uploadthing.com): Used for storing PDFs
* [OpenAI](https://openai.com/): Used to generate embeddings of the uploaded documents and for the chat itself
* [Stripe](https://stripe.com/): This is optional: The application has support for limited free tier and a powerful paid tier. You can learn more about [integrating Nile and Stripe in the integration guide](https://www.thenile.dev/docs/platform/integrations/stripe).
### 6. Setting the environment
* If you haven't cloned this project yet, now will be an excellent time to do so. Since it uses NextJS, we can use `create-next-app` for this:
```bash theme={null}
npx create-next-app -e https://github.com/niledatabase/niledatabase/tree/main/examples/ai/ai_pdf nile-ai-pdf
cd nile-ai-pdf
```
* Rename `.env.example` to `.env.local`, and update it with your workspace and database name.
*(Your workspace and database name are displayed in the header of the Nile dashboard.)*
Fill in the username and password with the credentials you picked up in the previous step.
And fill in the access keys for UploadThing and OpenAI.
* Install dependencies with `yarn install` or `npm install`.
# Autonomous Code Assistant - Code more, type less
Source: https://thenile.dev/docs/getting-started/examples/code_assistant
In this tutorial, you will learn how to use Nile's tenant virtualization, user management and [vector embedding](/ai-embeddings/vectors/pg_vector) features to build a
SaaS application that helps browse and query new code-bases.
It also uses LangChain SDK and OpenAI's APIs with Nile's SDK to implement a [RAG architecture](/ai-embeddings/rag) where the input text is a question in english, the retrieved documents are relevant code snippets and the response includes both
an answer and the code snippets.
Embeddings for the code in each repository are generated with OpenAI's `text-embedding-3-large model` and stored in Nile. The code snippets are stored in a separate table in Nile.
And the response to each question is generated by querying the code snippets table using pg\_vector extension, and then sending the relevant documents to
OpenAI's `gpt-4o-mini` model. The response is [streamed back to the user](https://www.thenile.dev/docs/ai-embeddings/streaming) in real-time.
Because of Nile's virtual tenant databases, the retrieved code snippets will only be from the tenant the user selected,
and Nile validates that the user has permissions to view this tenant's data. No risk of accidentally retrieving code that belongs to the wrong tenant.
### 1. Create a database
1. Sign up for an invite to [Nile](https://thenile.dev) if you don't have one already
2. You should see a welcome message. Click on "Lets get started"
3. Give your workspace and database names, or you can accept the default auto-generated names.
### 2. Create tables
After you created a database, you will land in Nile's query editor. Create the following table for storing our embeddings:
```sql theme={null}
CREATE TABLE IF NOT EXISTS embeddings_openai_text3_large (
tenant_id UUID ,
id UUID DEFAULT gen_random_uuid (),
file_id UUID,
embedding vector(1024) NOT NULL,
primary key (tenant_id, id)
);
```
Note the `embedding` column is of type `vector(1024)`. Vector type is provided by the pg\_vector extension for storing embeddings.
The size of the vector has to match the number of dimensions in the model you use.
The table has `tenant_id` column, which makes it tenant aware. By storing embeddings in a tenant-aware table, we can use Nile's built-in tenant isolation to ensure that
information about PDFs won't leak between tenants.
We also need somewhere to store the code itself. It doesn't have to be Postgres - S3 or Github are fine. But Postgres is convenient in our example.
```sql theme={null}
CREATE TABLE IF NOT EXISTS file_content (
tenant_id UUID ,
project_id UUID,
id UUID DEFAULT gen_random_uuid (),
file_name VARCHAR(255) NOT NULL,
contents TEXT NOT NULL,
primary key (tenant_id, project_id, id)
);
```
and to store information about each project:
```sql theme={null}
CREATE TABLE IF NOT EXISTS projects (
tenant_id UUID,
id UUID DEFAULT gen_random_uuid (),
name varchar(30),
url varchar(1024),
description TEXT,
primary key (tenant_id, id)
)
```
If all went well, you'll see the new tables in the panel on the left hand side of the query editor. You can also see Nile's built-in tenant table next to it.
You can also explore the schema in the "Schema Visualizer"
### 3. Getting Nile credentials
In the left-hand menu, click on "Settings" and then select "Credentials". Generate credentials and keep them somewhere safe. These give you access to the database.
In addition, you'll need the API URL. You'll find it under "Settings" in the "General" page.
### 4. Third party dependencies
This project uses OpenAI for both embeddings and chat models. To run this example, you will need an OpenAI API keys.
This demo uses Google authentication for signup. You will need to configure this in both Google and Nile,
following the instructions [in the example](https://github.com/niledatabase/niledatabase/blob/main/examples/user_management/social_login_google/NextJS/README.md).
### 5. Setting the environment
* If you haven't cloned this project yet, now will be an excellent time to do so. Since it uses NextJS, we can use `create-next-app` for this:
```bash theme={null}
npx create-next-app -e https://github.com/niledatabase/niledatabase/tree/main/examples/ai/code_assist code_assist
cd code_assist
```
* Rename `.env.example` to `.env.local`, and update it with your Nile credentials, OpenAI credentials and (if using Google SSO) Nile's API URL.
* Install dependencies with `npm install`.
### 6. Generating Embeddings
You'll need to start by generating and storing embeddings for a few interesting projects.
* Start by cloning some interesting repos to your laptop. You can start with this examples repository, but any repo will work.
* Open `src/lib/OrgRepoEmbedder.ts`
* Edit the `await embedDirectory(...)'` calls to refer to your repos. It is typical to map each github organization to a Nile tenant and each repo to a project, but you can model this in any way that makes sense to you. Keep in mind that CodeAssist will only use embeddings from the current project and current tenant as context, so make sure each project is interesting enough to discuss.
* Run the embedder with `node --experimental-specifier-resolution=node --loader ts-node/esm --no-warnings src/lib/OrgRepoEmbedder.ts`
* The embedder automatically creates tenants, projects and embeddings for you. You can use Nile Console to view the data you just generated and double check that it all looks right.
### 7. Running the app
To run the app, simply:
```bash theme={null}
npm run dev
```
Open [http://localhost:3000](http://localhost:3000) with your browser to see the result.
If all went well, your browser should show you the first page in the app, asking you to login or sign up.
After you sign up as a user of this example app, you'll be able to see this user by going back to Nile Console and running `select * from users` in the query editor.
Once you choose a tenant, you can select a project, browse files, and most important - ask our CodeAssist any question about the projects you embedded.
## Learn More
To learn more about how this example works and how to use Nile:
* [More about AI and embeddings in Nile](https://www.thenile.dev/docs/ai-embeddings)
* [More about tenants in Nile](https://www.thenile.dev/docs/tenant-virtualization/tenant-management)
* [Nile's Javascript SDK reference](https://www.thenile.dev/docs/reference/sdk-reference)
## Deploy on Vercel
The easiest way to deploy your Next.js app is to use the [Vercel Platform](https://vercel.com/new?utm_medium=default-template\&filter=next.js\&utm_source=create-next-app\&utm_campaign=create-next-app-readme) from the creators of Next.js.
Check out our [Next.js deployment documentation](https://nextjs.org/docs/deployment) for more details.
# Sales Insight - AI-native, multi-tenant enterprise sales assistant
Source: https://thenile.dev/docs/getting-started/examples/sales_insight
In this tutorial, you will learn how to use Nile's tenant virtualization and [vector embedding](/ai-embeddings/vectors/pg_vector) to build a
SaaS application that helps browse and query new code-bases.
The web application we'll build is designed to be end-to-end serverless [RAG architecture](/ai-embeddings/rag). We'll use Modal to deploy the web application and also deploy a
Llama 3.1 7B model. We'll use Nile's serverless Postgres to store the embeddings and the application data and retrieve the relevant parts of the sales transcripts.
Embeddings for the code in each repository are generated with OpenAI's `text-embedding-3-small` model and stored in Nile with the relevant chunks of the transcripts.
The response to each question is generated by querying the code snippets table using pg\_vector extension, and then sending the relevant documents to
the Llama 3.1 7B model, deployed on A100 GPU from Modal. The response is [streamed back to the user](https://www.thenile.dev/docs/ai-embeddings/streaming) in real-time.
Because of Nile's virtual tenant databases, the retrieved transcripts will only be from the tenant the user selected.
Nile validates that the user has permissions to view this tenant's data. No risk of accidentally retrieving code that belongs to the wrong tenant.
### 1. Create a database
1. Sign up for a [Nile account](https://console.thenile.dev) if you don't have one already
2. You should see a welcome message. Click on "Lets get started"
3. Give your workspace and database names, or you can accept the default auto-generated names.
### 2. Create tables
After you created a database, you will land in Nile's query editor. Create the following table for storing our embeddings:
```sql theme={null}
create table call_chunks (
tenant_id uuid,
conversation_id varchar(50),
chunk_id int,
speaker_role varchar(20),
content text,
embedding vector(1536) -- must match the embedding model dimensions
);
```
Note the `embedding` column is of type `vector(1536)`. Vector type is provided by the pg\_vector extension for storing embeddings.
The size of the vector has to **exactly** match the number of dimensions in the model you use.
The table has `tenant_id` column, which makes it tenant aware. By storing embeddings in a tenant-aware table, we can use Nile's built-in tenant
isolation to ensure that sales transcripts won't leak between tenants.
If all went well, you'll see the new table in the panel on the left hand side of the query editor. You can also see Nile's built-in tenant table next to it.
You can also explore the schema in the "Schema Visualizer"
### 3. Getting Nile credentials
In the left-hand menu, click on "Settings" and then select "Credentials". Generate credentials and keep them somewhere safe. These give you access to the database.
In addition, you'll need the API URL. You'll find it under "Settings" in the "General" page.
### 4. Third party dependencies
This project uses Modal for hosting the web application and the LLM. You will need to have a Modal account, set up the Modal CLI,
nd link your Nile credentials to your Modal account. Our [Modal Integration doc](/integrations/modal) will help you through this.
We are also using [Hugging Face](https://huggingface.co/) for the Llama 3.1 7B model. You will need to have a Hugging Face account for this, and an HuggingFace API key.
Set up the API key as a secret in Modal using Hugging Face integration.
We are using OpenAI to generate embeddings for sales transcripts and questions.
You will need to have an OpenAI account and an OpenAI API key. Store the OpenAI key as a secret in Modal. The way this example is set up, you'll want to create a secret called "embedding-config"
and store two keys in it: `OPENAI_API_KEY` and `EMBEDDING_MODEL` (e.g. `text-embedding-3-small`).
### 5. Setting the environment
* If you haven't cloned this project yet, now will be an excellent time to do so. Since it uses NextJS, we can use `create-next-app` for this:
```bash theme={null}
git clone https://github.com/niledatabase/niledatabase.git
cd niledatabase/examples/sales_insight
```
* Create a `.env` file and add your Nile connection string:
```bash theme={null}
DATABASE_URL=postgresql://user:password@us-west-2.db.thenile.dev:5432/mydb
```
* Install dependencies with `pip install -r requirements.txt`
### 6. Loading data
We prepared example data, that is already chunked and embedded. This will let you start faster and save on OpenAI credits.
To get the prepared data, you need to:
* Install git-lfs. You can download from their website or "brew install git-lfs"
* `git lfs install`
* Upload your SSH public key to your Hugging Face account.
* Configure `ssh` to use the key you just uploaded to Hugging Face when accessing the repo. You'll need to put this snippet in your `.ssh/config` file:
```
Host hf.co
AddKeysToAgent yes
UseKeychain yes
IdentityFile ~/.ssh/myhfkey
```
* Clone the repo with: git clone [https://huggingface.co/datasets/gwenshap/sales-transcripts/resolve/main/README.md](https://huggingface.co/datasets/gwenshap/sales-transcripts/resolve/main/README.md)
* Copy the data directory to the root of the example app: `cp -r sales-transcripts/data .`
Now, you can load the data by running:
```bash theme={null}
python -m ingest.load_embeddings
```
### 7. Download the Llama 3.1 7B model to Modal
This step is important, as it will take a while to download the model and its weights, and you don't want to do it on every deployment.
```bash theme={null}
modal run download_llama.py
```
Note that Llama models require approval from Meta before downloading.
You can request access to the Llama 3.1 7B model by clicking on the "Request Access" button on the model page in Hugging Face.
At the end of the process, you'll have a volume with the model and its weights. We'll use this volume in the llm container.
### 8. Building the UI
The example has a React frontend. We pre-build it with Vite and then serve the static files in the Modal app.
```bash theme={null}
cd ui
npm install
npm run build
cd ..
```
### 9. Running the app
You run the app in Modal. We will start by running it in `dev` mode, which will start a modal app and a web server.
This will let you see the logs in the console, and it will automatically re-deploy if you make changes.
```bash theme={null}
modal serve web_app.py
```
If all went well, modal will print a URL. Go this URL in your browser to see the app.
Your browser should show you the first page in the app, asking you to login or sign up.
After you sign up as a user of this example app, you'll be able to see this user by going back to Nile Console and running `select * from users` in the query editor.
Once you choose a tenant, you select a sales transcript and can ask questions about the transcript.
## Learn More
To learn more about how this example works and how to use Nile:
* [More about AI and embeddings in Nile](https://www.thenile.dev/docs/ai-embeddings)
* [More about tenants in Nile](https://www.thenile.dev/docs/tenant-virtualization/tenant-management)
# Task Genius - AI-native, multi-tenant enterprise task manager
Source: https://thenile.dev/docs/getting-started/examples/task_genius
In this tutorial, you will learn to build a multi-tenant AI-native todo list application, using Nile with Python, FastAPI, SQLAlchemy, and OpenAI's client.
We'll use Nile to provide us with virtual-tenant databases - isolating the tasks for each tenant, and we'll use the AI models to generate automated time estimates
for each task in the todo list.
### 1. Create a database
1. Sign up for an invite to [Nile](https://thenile.dev) if you don't have one already
2. You should see a welcome message. Click on "Lets get started"
3. Give your workspace and database names, or you can accept the default auto-generated names. In order to complete this quickstart in a browser, make sure you select to “Use Token in Browser”.
### 2. Create a table
After you created a database, you will land in Nile's query editor.
For our todo list application, we'll need tables to store tenants, users and todos.
Tenants and users already exists in Nile, they are built-in tables and you can see them in the list on the left side of the screen.
We'll just need to create a table for todos.
```sql theme={null}
create table todos (
id uuid DEFAULT (gen_random_uuid()),
tenant_id uuid,
title varchar(256),
estimate varchar(256),
embedding vector(768),
complete boolean);
```
You will see the new table in the panel on the left side of the screen, and you can expand it to view the columns.
The embedding column is a vector representation of the task. When the user adds new tasks, we will use these embeddings to find
semantically related tasks and use this as a basis of our AI-driven time estimates. This technique - looking up related data using embeddings and
using this data with text generation models is called [RAG (Retrieval Augumented Generation)](https://www.thenile.dev/docs/ai-embeddings/rag).
See the `tenant_id` column? By specifying this column, You are making the table **tenant aware**. The rows in it will belong to specific tenants. If you leave it out, the table is considered shared, more on this later.
### 3. Get credentials
In the left-hand menu, click on "Settings" and then select "Credentials". Generate credentials and keep them somewhere safe. These give you access to the database.
### 4. 3rd party credentials
This example uses AI chat and embedding models to generate automated time estimates for each task in the todo list. In order to use this functionality, you will need access to models from a vendor with OpenAI compatible APIs. Make sure you have an API key, API base URL and the [names of the models you'll want to use](https://www.thenile.dev/docs/ai-embeddings/embedding_models).
### 5. Set the environment
Enough GUI for now. Let's get to some code.
If you haven't cloned this repository yet, now will be an excellent time to do so.
```bash theme={null}
git clone https://github.com/niledatabase/niledatabase
cd niledatabase/examples/quickstart/python
```
Copy `.env.example` to `.env` and fill in the details of your Nile DB. The ones you copied and kept safe in step 3.
It should look something like this:
```bash theme={null}
DATABASE_URL=postgresql://user:password@db.thenile.dev:5432/mydb
LOG_LEVEL=DEBUG
SECRET_KEY = "09d25e094faa6ca2556c818166b7a9563b93f7099f6f0f4caa6cf63b88e8d3e7"
ALGORITHM = "HS256"
ACCESS_TOKEN_EXPIRE_MINUTES = 30
# for AI estimates
AI_API_KEY=your_api_key_for_openai_compatible_service
AI_BASE_URL=https://api.fireworks.ai/inference/v1
AI_MODEL=accounts/fireworks/models/llama-v3p1-405b-instruct
EMBEDDING_MODEL=nomic-ai/nomic-embed-text-v1.5
```
Optional, but recommended, step is to set up a virtual Python environment:
```bash theme={null}
python -m venv venv
source venv/bin/activate
```
Then, install dependencies:
```bash theme={null}
pip install -r requirements.txt
```
### 5. Run the application
If you'd like to use the app with the UI, you'll want to build the UI assets first:
```bash theme={null}
cd ui
npm install
npm run build
```
Then start the Python webapp:
```bash theme={null}
uvicorn main:app --reload
```
Go to [http://localhost:8000](http://localhost:8000) in a browser to see the app.
You can try a few things in the app:
* Sign up as a new user
* Create a tenant
* Create a todo task and see its time estimate. If you create more tasks, the estimates for new tasks will use the embeddings of the existing tasks to generate the estimates.
You can also use the API directly:
```bash theme={null}
# login
curl -c cookies -X POST 'http://localhost:8000/api/login' \
--header 'Content-Type: application/json' \
--data-raw '{"email":"test9@pytest.org","password":"foobar"}'
# create tenant
curl -b cookies -X POST 'localhost:8000/api/tenants' \
--header 'Content-Type: application/json' \
--data-raw '{"name":"my first customer"}'
# list tenants
curl -b cookies -X GET 'http://localhost:8000/api/tenants'
# create todo for a tenant (make sure you replace the tenant ID with the one you got from the previous step)
curl -b cookies -X POST \
'http://localhost:8000/api/todos' \
--header 'Content-Type: application/json' \
--header 'X-Tenant-Id: 3c9bfcd0-7702-4e0e-b3f0-4e84221e20a7' \
--data-raw '{"title": "feed the cat", "complete": false}'
# list todos for a tenant (make sure you replace the tenant ID with the one you got from the previous step)
curl -b cookies -X GET \
--header 'X-Tenant-Id: 3c9bfcd0-7702-4e0e-b3f0-4e84221e20a7' \
'http://localhost:8000/api/todos'
```
### 6. Check the data in Nile
Go back to the Nile query editor and see the data you created from the app.
```sql theme={null}
SELECT tenants.name, title, estimate, complete
FROM todos join tenants on tenants.id = todos.tenant_id;
```
You should see all the todos you created, and the tenants they belong to.
### 7. How does it work?
The app uses FastAPI, a modern Python web framework, and SQLAlchemy, a popular ORM. The app is built with tenants in mind, and it uses Nile's tenant context to isolate data between tenants.
`main.py` is the entry point of the app. It sets up the FastAPI app, registers the middleware and has all the routes.
### 7.1 Using AI models for time estimates
This example uses AI chat and embedding models to generate automated time estimates for each task in the todo list. We handle the time estimates in
the `create_todo` method which is the route handler for `@app.post("/api/todos")`. This handler executes when users add new tasks.
This is what the handler code looks like:
```python theme={null}
similar_tasks = get_similar_tasks(session, todo.title)
logger.info(f"Generating estimate based on similar tasks: {similar_tasks}")
estimate = ai_estimate(todo.title, similar_tasks)
embedding = get_embedding(todo.title, EmbeddingTasks.SEARCH_DOCUMENT)
todo.embedding = embedding
todo.estimate = estimate
session.add(todo)
session.commit()
```
As you can see, we look up similar tasks and then use the AI model to generate the estimate. We then store the task, with the estimate and the task embedding in the database.
The stored embedding will be used to find similar tasks in the future. The methods `get_similar_tasks`, `ai_estimate` and `get_embedding` are all defined in `ai_utils.py`.
They are wrappers around standard AI model calls and database queries, and they handle the specifics of the AI model we are using.
This will make it easy to switch models in the future.
Getting similar tasks is done by querying the database for tasks with similar embeddings. Before we search the database, we need to generate the embedding for the new task:
```python theme={null}
def get_similar_tasks(session: any, text: str):
query_embedding = get_embedding(text, EmbeddingTasks.SEARCH_QUERY)
similar_tasks_raw = (
session.query(Todo)
.filter(Todo.embedding.cosine_distance(query_embedding) < 1)
.order_by(Todo.embedding.cosine_distance(query_embedding)).limit(3))
return [{"title": task.title, "estimate": task.estimate} for task in similar_tasks_raw]
```
We started by generating an embedding with `SEARCH_QUERY` task type. This is because we are looking for similar tasks to the new task. We use an embedding model
from the `nomic` family, which is trained to perform specific types of embedding tasks. Telling it that we are generating the embedding for a lookup vs
generating an embedding that we will store with the document (as we'll do in a bit), should help the model produce more relevant results.
In order to use vector embeddings with SQL Alchemy and SQL Model ORM, we used [PG Vector's Python library](https://github.com/pgvector/pgvector-python).
You'll find it in `requirements.txt` for the project. Note that we filter out results where the cosine distance is higher than 1.
The lower the cosine distance is, the more similar the tasks are (0 indicate that they are identical).
A cosine distance of 1 means that the vectors are essentially unrelated, and when cosine distance is closer to 2, it indicates that the vectors are semantically opposites.
The `get_embedding` function uses the embedding model to generate the embedding and is a very simple wrapper on the model:
```python theme={null}
response = client.embeddings.create(
model=os.getenv("EMBEDDING_MODEL"),
input=adjust_input(text, task),
)
```
Now that we have the similar tasks, the handler calls `ai_estimate` to generate the time estimate.
This function also wraps a model, this time a conversation model rather than an embedding model. And it icludes the similar tasks in the promopt, so the model will
generate similar estimates:
```python theme={null}
response = client.chat.completions.create(
model = os.getenv("AI_MODEL"),
messages = [
{
"role": "user",
"content" :
f'you are an amazing project manager. I need to {text}. How long do you think this will take?'
f'I have a few similar tasks with their estimates, please use them as reference: {similar_tasks}.'
f'respond with just the estimate, no yapping.',
},
],
)
```
This estimate is then stored in the database along with the task and its vector embedding.
### 7.2 Working with virtual tenant databases
The first thing we do in the app is set up the tenant middleware. The `TenantAwareMiddleware` is defined in `middleware.py`,
it is a simple middleware that reads the `X-Tenant-Id` header and sets the tenant context for the request. This is how we know which tenant the request is for.
```python theme={null}
app = FastAPI()
app.add_middleware(TenantAwareMiddleware)
```
The middleware runs before any request is processed. But not every request has a tenant context. For example, `login` or `create_tenant` routes doesn't need a tenant context.
Requests that don't have a tenant context are considered to be `global` since they are performed on the database as a whole, not in the virtual database for a specific tenant.
To handle a request in the global context, we use a global session. This is a session that doesn't have a tenant context. For example to create a new tenant:
```python theme={null}
@app.post("/api/tenants")
async def create_tenant(tenant:Tenant, request: Request, session = Depends(get_global_session)):
session.add(tenant)
session.commit()
return tenant
```
To handle a request in the tenant context, we use a tenant session. This is a session that has a tenant context. For example to list todos:
```python theme={null}
@app.get("/api/todos")
async def get_todos(session = Depends(get_tenant_session)):
todos = session.query(Todo).all()
return todos
```
This looks like it could return all todos from all tenants, but it doesn't. The `get_tenant_session`
function sets the tenant context for the session, and the query is executed in the virtual database of the tenant.
The last piece of the puzzle is the `get_tenant_session` function. It is defined in `db.py` and is responsible for creating the session with the correct context.
```python theme={null}
def get_tenant_session():
session = Session(bind=engine, expire_on_commit=False)
try:
tenant_id = get_tenant_id()
user_id = get_user_id()
session.execute(text(f"SET nile.tenant_id='{tenant_id}';"))
session.execute(text(f"SET nile.user_id='{user_id}';"))
yield session
except:
session.rollback()
raise
finally:
session.execute(text("RESET nile.user_id;"))
session.execute(text("RESET nile.tenant_id;"))
session.commit()
pass
```
We are setting both the user and tenant context in the session. This is important for security and isolation.
The user context is used to check if the user has access to the tenant, and the tenant context is used to isolate the data.
Note that we are using FastAPI dependency injection to get the session in the route handlers. This is a powerful feature of FastAPI that makes it easy to manage resources like sessions.
The `yield` keyword is used to return the session to the caller, and the `finally` block is used to clean up the session after the request is processed.
And this is it. Thats all we need to do to build a multi-tenant app with Nile, FastAPI and SQLAlchemy.
## 8. Looking good!
🏆 Tada! You have learned the basic Nile concepts:
* [Tenant aware tables](https://www.notion.so/640f7364152a4941990cf6351b065049?pvs=21)
* Tenant context
* Tenant isolation
# Introduction
Source: https://thenile.dev/docs/getting-started/getting-started
Nile - PostgreSQL re-engineered for B2B apps
Nile re-engineers PostgreSQL for multi-tenant apps, enabling to launch and scale quickly, securely, and in the most cost-effective manner. Nile can be used as a fully integrated solution or only in parts.
Nile's architecture includes storage/compute decoupling, page-level tenant isolation, global routing gateway and a database clustering technology to provide the following key benefits -
1. Built-in tenant virtualization in Postgres for better data and vector embedding isolation and performance isolation without any RLS or complex permission logic in the application across customers.
2. Vector embedding with RAG support for all model providers, cost-effective and auto-scales to billions of embeddings across customers
3. Use one database in your application but get tenant-level backups, customer/tenant insights, and schema management across tenants managed by Nile
4. Seamlessly autoscales as your customer's usage increases and scales to zero with no cold start time
5. Place some tenants/customers on dedicated compute in the same database and some in other regions. Nile lets you customize for cost, security, compliance, and latency for each tenant in your AI-native application without having to manage separate databases for each tenant
6. You can scale to millions of customers and billions of vector embeddings effortlessly as Nile horizontally scales
You can read more about Nile, its architecture and concepts in the [What is Nile](https://website-git-fixlanding-niledatabase.vercel.app/docs/getting-started/what_is_nile) section.
Choose your starting point to explore and try out Nile
## Languages
Nile is just Postgres. It works with all the tools and Postgres ecosystem. The quickstarts below covers the Javascript and Python ecosystem. You can also pick plain SQL if that is preferred.
{'file_type_pgsql'}
}
>
Get started with SQL
}
>
Get started with NextJS
}
>
Get started with Drizzle
{'file_type_prisma'}
}
>
Get started with Prisma
}
>
Get started with Python
}
>
Get started with Node
## Examples
These are end to end examples that are fully functional applications to show how easily you can build AI-native B2B applications with Nile. Feel free to fork the examples to learn more or play around with the hosted live demo.
}
/>
}
/>
}
/>
}
/>
## AI Embedding Models
Nile integrates with all the popular embedding models. Pick one and follow along to build an AI-native experience for your B2B application. Learn how you can build AI applications that are secure, performant and cost effective for each of your customers.
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
## Use cases
Nile is used by companies to build AI-native B2B applications. Here are some examples of how you can use Nile to build your next B2B application.
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
{'Notion icon'}
}
/>
}
/>
}
/>
{'001-business'}
}
/>
}
/>
}
/>
}
/>
## Schema Migrations
Nile supports schema migrations for your tenant aware tables. You can use your favorite migration tool to manage your schema migrations.
{'file_type_prisma'}
}
/>
}
/>
}
/>
}
/>
# Build AI-Native B2B application with Postgres and Drizzle ORM
Source: https://thenile.dev/docs/getting-started/languages/drizzle
In this tutorial, you will learn about Nile's tenant virtualization features, while building a backend service for a todo list application.
We'll use Drizzle as the ORM to interact with the database, OpenAI client to interact with AI models, Express as the web framework and NodeJS as the runtime.
We'll use Nile to provide us with virtual-tenant databases - isolating the tasks for each tenant, and we'll use the AI models to generate automated time estimates
for each task in the todo list. The estimates will be based on the task title, and estimates of similar tasks in the tenant's history.
This technique is known as [RAG architecture](https://www.thenile.dev/docs/ai-embeddings/rag).
1. Sign up for an invite to [Nile](https://thenile.dev) if you don't have one already
2. You should see a welcome message. Click on "Lets get started"
3. Give your workspace and database names, or you can accept the default auto-generated names. In order to complete this quickstart in a browser, make sure you select to “Use Token in Browser”.
After you created a database, you will land in Nile's query editor. Since our application requires a table for storing all the "todos" this is a good time to create one:
```sql theme={null}
CREATE TABLE IF NOT EXISTS "todos" (
"id" uuid DEFAULT gen_random_uuid(),
"tenant_id" uuid,
"title" varchar(256),
"estimate" varchar(256),
"embedding" vector(768),
"complete" boolean,
CONSTRAINT todos_tenant_id_id PRIMARY KEY("tenant_id","id")
);
```
You will see the new table in the panel on the left side of the screen, and you can expand it to view the columns.
The embedding column is a vector representation of the task. When the user adds new tasks, we will use these embeddings to find
semantically related tasks and use this as a basis of our AI-driven time estimates. This technique - looking up related data using embeddings and
using this data with text generation models is a key part of [RAG (Retrieval Augumented Generation)](https://www.thenile.dev/docs/ai-embeddings/rag).
See the `tenant_id` column? By specifying this column, You are making the table **tenant aware**. The rows in it will belong to specific tenants. If you leave it out, the table is considered shared, more on this later.
In the left-hand menu, click on "Settings" and then select "Connection".
Click on the Postgres button, then click "Generate Credentials" on the top right corner. Copy the connection string - it should now contain the credentials we just generated.
This example uses AI chat and embedding models to generate automated time estimates for each task in the todo
list. In order to use this functionality, you will need access to models from a vendor with OpenAI compatible
APIs. Make sure you have an API key, API base URL and the [names of the models you'll want to use](https://www.thenile.dev/docs/ai-embeddings/embedding_models).
Enough GUI for now. Let's get to some code.
If you haven't cloned this repository yet, now will be an excellent time to do so.
```bash theme={null}
git clone https://github.com/niledatabase/niledatabase
cd niledatabase/examples/quickstart/drizzle
```
Rename `.env.example` to `.env`, and update it with the connection string you just
copied from Nile Console and the configuration of your AI vendor and model. Make sure you don't include
the word "psql". It should look something like this:
```bash theme={null}
DATABASE_URL=postgres://018b778a-30df-7cdd-b55c-2f9664db39f3:ff3fb983-683c-4616-bbbc-519d8ddbbce5@db.thenile.dev:5432/gwen_db
# for AI estimates
AI_API_KEY=your_api_key_for_openai_compatible_service
AI_BASE_URL=https://api.fireworks.ai/inference/v1
AI_MODEL=accounts/fireworks/models/llama-v3p1-405b-instruct
EMBEDDING_MODEL=nomic-ai/nomic-embed-text-v1.5
```
Install dependencies with `yarn install` or `npm install`.
```bash theme={null}
npm install
```
Start the web service with `npm start` or `yarn start`.
Now you can use `curl` to explore the APIs. Here are a few examples:
```bash theme={null}
# create a tenant
curl --location --request POST 'localhost:3001/api/tenants' \
--header 'Content-Type: application/json' \
--data-raw '{"name":"my first customer", "id":"108124a5-2e34-418a-9735-b93082e9fbf2"}'
# get tenants
curl -X GET 'http://localhost:3001/api/tenants'
# create a todo (don't forget to use a read tenant-id in the URL)
curl -X POST \
'http://localhost:3001/api/tenants/108124a5-2e34-418a-9735-b93082e9fbf2/todos' \
--header 'Content-Type: application/json' \
--data-raw '{"title": "feed the cat", "complete": false}'
# list todos for tenant (don't forget to use a read tenant-id in the URL)
curl -X GET \
'http://localhost:3001/api/tenants/108124a5-2e34-418a-9735-b93082e9fbf2/todos'
# list todos for all tenants
curl -X GET \
'http://localhost:3001/insecure/all_todos'
```
Go back to the Nile query editor and see the data you created from the app.
```sql theme={null}
SELECT tenants.name, title, estimate, complete
FROM todos join tenants on tenants.id = todos.tenant_id;
```
You should see all the todos you created, and the tenants they belong to.
If you create more tasks via the REST APIs, the estimates for new tasks will use the embeddings of the existing tasks to
generate the estimates.
This example is a good starting point for building your own application with Nile.
You have learned basic Nile, AI and RAG concepts and how to use them with Drizzle ORM.
You can learn how the example application works more in detail below or you can learn
more about Nile's tenant virtualization features in the following tutorials:
* [Tenant management](/tenant-virtualization/tenant-management)
* [Tenant isolation](/tenant-virtualization/tenant-isolation)
You can explore Nile's JS SDK in the [SDK reference](/auth/sdk-reference/javascript/overview).
You can learn [More about AI in Nile](https://www.thenile.dev/docs/ai-embeddings), or try a more advanced example like:
* [Chat with PDFs](https://www.thenile.dev/docs/getting-started/examples/chat_with_pdf)
* [Code Assistnat](https://www.thenile.dev/docs/getting-started/examples/code_assistant)
## How does it work?
Let's take a look at the code. The web application starting point is in [`src/app.ts`](https://github.com/niledatabase/niledatabase/blob/main/examples/quickstart/drizzle/src/app.ts).
It creates an Express app, and sets up some of the usual middleware:
```typescript theme={null}
const app = express();
app.listen(PORT, () => console.log(`Server is running on port ${PORT}`));
app.use(express.json());
app.use(express.urlencoded({ extended: true }));
```
The interesting parts are the middleware that sets up the tenant context, and `tenantDB` which wraps Drizzle and passes the tenant context to Nile.
And of course, the handler that uses AI models to automatically estimate the time to complete tasks.
Lets look at these one by one.
### Tenant Context Middleware
Immediately after the middleware that sets up the Express app, we add a middleware that introduces Nile's tenant context to the application:
```typescript theme={null}
app.use((req, res, next) => {
const fn = match('/api/tenants/:tenantId/todos', {
decode: decodeURIComponent,
});
const m = fn(req.path);
//@ts-ignore
const tenantId = m?.params?.tenantId;
console.log('setting context to tenant: ' + tenantId);
tenantContext.run(tenantId, next);
});
```
This may look a bit magical, but let me explain. This middleware grabs the tenant id from the URL and sets it in the tenant context.
This context is an instance of `AsyncLocalStorage` - which provides a way to store data that is global and unique per execution context.
The data we are storing is the tenant ID, and by passing `next` as the second argument to `tenantContext.run` we are making sure that all the code that runs after this middleware will have access to the tenant ID.
### TenantDB
Now lets take a look at `db/db.ts` - this is where we set up Drizzle and connect it to Nile. We use `pg` client as the driver for Drizzle, and we pass it the connection string from the environment.
```typescript theme={null}
const client = new Client({
connectionString: process.env.DATABASE_URL,
});
// Drizzle expects the connection to be open when using a client. Alternatively, you can use a pool
await client.connect();
// check the connection
const res = await client.query('SELECT $1::text as message', [
'Client connected to Nile',
]);
console.log(res.rows[0].message);
export const db = drizzle(client, { logger: true });
```
But the interesting part is the `tenantDB` function. This is where we wrap Drizzle with Nile's tenant virtualization features.
We need Drizzle to set Nile context on the connection before each query we execute. We do this by accepting each query as a callback and wrapping it in a transaction that starts with `set nile.tenant_id`.
We get the tenant ID from the tenant context, which we set earlier in the middleware.
```typescript theme={null}
return db.transaction(async (tx) => {
const tenantId = tenantContext.getStore();
// clean old context
await tx.execute(sql`reset nile.tenant_id`);
// if there's a tenant ID, set it in the context
if (tenantId) {
await tx.execute(sql`set nile.tenant_id = '${sql.raw(tenantId)}'`);
}
return cb(tx);
});
```
### Tying it all together - handling a request for all todos for a tenant
Now that we understand how the tenant context and `tenantDB` work, lets take a look at how we use them to handle a request for all todos for a tenant.
We are using the `tenantDB` function to execute a query that returns all the todos for a tenant.
The query doesn't need to include the tenant ID in the `where` clause, because `tenantDB` will set it in the context.
```typescript theme={null}
const todos = await tenantDB(async (tx) => {
return await tx.select().from(todoSchema);
});
res.json(todos);
```
### Using AI models to estimate time to complete tasks
This example uses AI chat and embedding models to generate automated time estimates for each task in the todo list. We handle the time estimates in
the route handler for `app.post("/api/tenants/:tenantId/todos"`. This handler executes when users add new tasks.
This is what the handler code looks like:
```typescript theme={null}
const { title, complete } = req.body;
const tenantId = req.params.tenantId;
// We are using tenantDB with tenant context to ensure that we only find tasks for the current tenant
const similarTasks = await findSimilarTasks(tenantDB, title);
console.log('found similar tasks: ' + JSON.stringify(similarTasks));
const estimate = await aiEstimate(title, similarTasks);
console.log('estimated time: ' + estimate);
// get the embedding for the task, so we can find it in future similarity searches
const embedding = await embedTask(title, EmbeddingTasks.SEARCH_DOCUMENT);
const newTodo = await tenantDB(async (tx) => {
return await tx
.insert(todoSchema)
.values({ tenantId, title, complete, estimate, embedding })
.returning();
});
// return without the embedding vector, since it is huge and useless
res.json(newTodo.map((t: any) => ({ ...t, embedding: '' })));
```
As you can see, we look up similar tasks and then use the AI model to generate the estimate. We then store the task, with the estimate and the task embedding in the database.
The stored embedding will be used to find similar tasks in the future. The methods `findSimilarTasks`, `aiEstimate` and `embedTask` are all defined in `AiUtils.ts`.
They are wrappers around standard AI model calls and database queries, and they handle the specifics of the AI model we are using.
This will make it easy to switch models in the future.
Getting similar tasks is done by querying the database for tasks with similar embeddings.
```typescript theme={null}
const embedding = await embedTask(title, EmbeddingTasks.SEARCH_QUERY);
const similarity = sql`(${cosineDistance(
todoSchema.embedding,
embedding,
)})`;
// get similar tasks, no need to filter by tenant because we are already in the tenant context
const similarTasks = await tenantNile(async (tx) => {
return await tx
.select({ task: todoSchema.title, estimate: todoSchema.estimate })
.from(todoSchema)
.where(lt(similarity, 1))
.orderBy((t: any) => desc(similarity))
.limit(3);
});
```
We started by generating an embedding with `SEARCH_QUERY` task type. This is because we are looking for
similar tasks to the new task. We use an embedding model from the `nomic` family, which is trained to perform specific types of embedding tasks. Telling it that we are generating the embedding for a lookup vs
generating an embedding that we will store with the document (as we'll do in a bit), should help the model produce more relevant results.
DrizzleORM provides the magical `sql` method, as well as the `cosineDistance` function, which we use to work
with `pg_vector` and calculate the similarity between the embeddings of the new task and the existing tasks.
As you can see, we filter out results where the cosine distance is higher than 1.
The lower the cosine distance is, the more similar the tasks are (0 indicate that they are identical).
A cosine distance of 1 means that the vectors are essentially unrelated, and when cosine distance is closer to 2, it indicates that the vectors are semantically opposites.
The `embedTask` function uses the embedding model to generate the embedding and is a very simple wrapper on the model:
```typescript theme={null}
let resp = await ai.embeddings.create({
model: EMBEDDING_MODEL,
input: adjust_input(title, task),
});
```
Now that we have the similar tasks, the handler calls `aiEstimate` to generate the time estimate.
This function also wraps a model, this time a conversation model rather than an embedding model. And it icludes the similar tasks in the promopt, so the model will
generate similar estimates:
```typescript theme={null}
const aiEstimate = await ai.chat.completions.create({
messages: [
{
role: 'user',
content: `you are an amazing project manager. I need to ${title}. How long do you think this will take?
I have a few similar tasks with their estimates, please use them as reference: ${similarTasks}.
respond with just the estimate, keep the answer short.`,
},
],
max_tokens: 64, // limit the response to 64 tokens
model: model,
});
```
This estimate is then stored in the database along with the task and its vector embedding.
### But wait, what's todoSchema and tenantSchema?
You may have noticed that we are not using the table name directly in the code. Instead, we are using `todoSchema` and `tenantSchema`.
We defined these in `src/db/schema.ts`, and these are the objects that Drizzle uses to interact with the database. It is an object that maps to the tables in our database.
Drizzle can also generate migration files from these objects, and we can use them to create the tables in the database.
# Build a B2B Application with Postgres and NextJS
Source: https://thenile.dev/docs/getting-started/languages/nextjs
In this quick tutorial, you will learn about Nile's tenant virtualization features, while building a todo list application with NextJS 13.
1. Go to the [Nile console](https://console.thenile.dev) and create a workspace and database if you have not done so already.
2. You should see a welcome message. Click on "Lets get started"
3. Give your workspace and database names, or you can accept the default auto-generated names.
After you created a database, you will land in Nile's query editor. Since our application requires a table for storing all the "todos" this is a good time to create one:
```sql theme={null}
create table todos (
id uuid DEFAULT (gen_random_uuid()),
tenant_id uuid,
title varchar(256),
estimate varchar(256),
complete boolean,
embedding vector(768)
);
```
You will see the new table in the panel on the left side of the screen, and you can expand it to view the columns.
See the `tenant_id` column? By specifying this column, You are making the table **tenant aware**. The rows in it will belong to specific tenants. If you leave it out, the table is considered shared, more on this later.
The embedding column is a vector representation of the task. When the user adds new tasks, we will use these embeddings to find semantically
related tasks and use this as a basis of our AI-driven time estimates. This technique - looking up related data using embeddings and using this data with text generation models is called [**RAG (Retrieval Augmented Generation)**](https://www.thenile.dev/docs/ai-embeddings/rag).
In the left-hand menu, click on "Settings" and then select "Credentials".
Generate credentials and keep them somewhere safe. These give you access to
the database.
This example uses AI chat and embedding models to generate automated time
estimates for each task in the todo list. In order to use this functionality,
you will need access to models from a vendor with OpenAI compatible APIs. Make
sure you have an API key, API base URL and the [names of the models you'll
want to use](https://www.thenile.dev/docs/ai-embeddings/embedding_models).
Enough GUI for now. Let's get to some code.
If you haven't cloned this repository yet, now will be an excellent time to do so.
```bash theme={null}
git clone https://github.com/niledatabase/niledatabase
cd niledatabase/examples/quickstart/nextjs
```
Rename `.env.local.example` to `.env.local`, and update it with your database credentials.
It should look something like this (you can see that I used Fireworks as the vendor, but you can use OpenAI or any compatible vendor):
```bash theme={null}
# Private env vars that should never show up in the browser
# These are used by the server to connect to Nile database
NILEDB_USER=018ad484-0d52-7274-8639-057814be60c3
NILEDB_PASSWORD=0d11b8e5-fbbc-4639-be44-8ab72947ec5b
AI_API_KEY=your-ai-vendor-api-key
AI_BASE_URL=https://api.fireworks.ai/inference/v1
AI_MODEL=accounts/fireworks/models/llama-v3p1-405b-instruct
EMBEDDING_MODEL=nomic-ai/nomic-embed-text-v1.5
```
Install dependencies
```bash theme={null}
npm install
```
```bash theme={null}
npm run dev
```
💡 Note: This example only works with Node 18 and above. You can check the version with `node -v`.
Open [http://localhost:3000](http://localhost:3000) with your browser to see the result.
If all went well, your browser should show you the first page in the app, asking you to login or sign up.
After you sign up as a user of this example app, you'll be able to see this user by going back to Nile Console and looking at the users table
```sql theme={null}
select * from users;
```
Login with the new user, and you can create a new tenant and add tasks for the tenant. You can see the changes in your Nile database by running
```sql theme={null}
select tenants.name, title, estimate, complete from
tenants join todos on tenants.id=todos.tenant_id;
```
This example is a good starting point for building your own application with Nile.
You have learned basic Nile concepts and how to use them with NextJS.
You can learn more about Nile's tenant virtualization features in the following tutorials:
* [Tenant management](/tenant-virtualization/tenant-management)
* [Tenant isolation](/tenant-virtualization/tenant-isolation)
You can explore Nile's JS SDK in the [SDK reference](/auth/sdk-reference/javascript/overview).
You can learn [More about AI in Nile](https://www.thenile.dev/docs/ai-embeddings), or try a more advanced example like:
* [Chat with PDFs](https://www.thenile.dev/docs/getting-started/examples/chat_with_pdf)
* [Code Assistant](https://www.thenile.dev/docs/getting-started/examples/code_assistant)
## How does it work?
There are a few moving pieces here, so let's break it down.
This example uses NextJS `app router`, so the application landing page is in `app/page.tsx`.
We'll start here and go over the code for creating tenants, selecting tenants and adding tasks.
The SDK uses your credentials to call the Nile API. Every page, route and function in our app can use the same `nile` instance to access Nile APIs and DB.
But, we need to make sure we are using the right user and tenant context.
So we call `configureNile`, which take the cookies from `next/headers` and the tenant ID.
After this point, we can use `nile` to access the database and APIs, and it will use the authenticated user and tenant context.
In addition to accessing the database and requesting data server side, there are also APIs that are integrated for you via [dynamic routes](https://nextjs.org/docs/pages/building-your-application/routing/dynamic-routes).
The route file `api/[...nile]/route.ts` intercepts routes from the client and responds accordingly. There is a single file that is used to export the nile instance into the routes, which is then used to export the various handlers.
In addition, the NextJS extension automatically handles obtaining the cookies from the request.
```typescript theme={null}
import { Nile } from '@niledatabase/server';
import { nextJs } from '@niledatabase/nextjs';
export const nile = Nile({ extensions: [nextJs] });
```
### Listing tenants
Next thing we do in `app/tenants/page.tsx` is to list all the tenants for the current user, but the SDK must know about the user. The simplest way to do that is to pass the NextJS cookies to the function.
We use Nile SDK to query the database for all the tenants for the current user.
```typescript theme={null}
const tenants = await ctx.tenants.list();
```
and then we render the list of tenants as a list of links:
```typescript theme={null}
{tenants.map((tenant: any) => (
{tenant.name}
))}
```
It looks a bit like magic, but [NextJS server fetching](https://nextjs.org/docs/app/building-your-application/data-fetching/fetching-caching-and-revalidating#fetching-data-on-the-server-with-third-party-libraries) and Nile SDK are integrated together to make this happen.
### Creating a tenant
When you click on the "Create Tenant" button, we pop out a modal form. The form is defined in `app/tenants/add-form.tsx` and uses [NextJS server actions](https://nextjs.org/docs/app/building-your-application/data-fetching/forms-and-mutations).
Create tenant logic is in `app/tenants/tenant-actions.tsx`.
In `createTenant` method, we use Nile SDK to create a new tenant and link it to the current user.
```typescript theme={null}
const createTenantResponse = await nile.tenants.create({
name: tenantName,
});
```
This call inserts a new tenant to `tenants` table, and also inserts a new row to `tenant_users` table, linking the current user to the new tenant.
You can see the new tenant in the Nile console by running `select * from tenants`.
Because the user is authenticated to the API, Nile automatically adds the user to the tenant they are creating.
### Listing tasks
When you create a tenant or select an existing tenant, you are taken to the tenant's todo list.
The code for this page is in `app/tenants/[tenantid]/todos/page.tsx`.
Since there is no API for listing specific data in your database, it is now time to being writing queries. Normally, you would write a `where` clause to accomplish some tenant-level isolation (and you still can), the Nile SDK does it a little bit differently.
Instead of using SQL `where` clauses to get only todos for this specific tenant, we are relying on Nile's tenant isolation feature:
```typescript theme={null}
const _nile = await nile.withContext({ tenantId });
// no need for where clause because we previously set Nile context
const todos = await _nile.query('select * from todos');
```
Note that this time we are configuring Nile with the tenant ID too. Previously called `configureNile` without parameters because there was no tenant yet.
Behind the scenes, Nile does two things:
* Check that the user is a member of the tenant and indeed has access to these todos
* Apply tenant isolation and execute the query in a "virtual tenant DB".
This also means that you don't need to implement the 3-way join between `todo`, `tenants` and `tenant_users`, so the code is simpler and more performant.
### Adding a task
When you click on the "+" to add a new task or on the checkbox to mark it as done, we again rely on NextJS server actions to make things simple.
You can find the logic for adding a task in `app/tenants/[tenantid]/todos/todo-actions.tsx`.
As you can see, we again use the tenant context to insert the new task:
```typescript theme={null}
const _nile = await nile.withContext({ tenantId });
await _nile.query(
`INSERT INTO todos (tenant_id, title, estimate, embedding, complete)
VALUES ($1, $2, $3, $4, $5)`,
[
tenantNile.tenantId,
title,
estimate?.substring(0, 255),
embeddingToSQL(embedding),
false,
],
);
```
or update an existing one:
```typescript theme={null}
const _nile = await nile.withContext({ tenantId });
await _nile.query(
`UPDATE todos
SET complete = $1
WHERE id = $2`,
[complete, id],
);
```
By setting the context, Nile will validate that the user indeed has permission to add tasks for this tenant,
and will make sure we only update tasks for the current tenant.
We don't want just anyone handing us tasks and telling us to do them, right?
### AI-driven time estimates
This example uses AI chat and embedding models to generate automated time estimates for each task in the todo list. We handle the time estimates in
`app/tenants/[tenantid]/todos/todo-actions.tsx`, when we add a new task.
When you add a new task, we use the embedding model to generate an embedding for the task text:
```typescript theme={null}
const embedding = await embedTask(title.toString());
```
The `embedTask` function is defined in `lib/AiUtils.tsx` and uses the embedding model to generate the embedding.
We wrap the call to the model since some vendors have slightly different inputs and outputs. This will let us switch vendors easily in the future.
```typescript theme={null}
// generate embeddings
let resp = await ai.embeddings.create({
model: embedding_model,
input: title,
});
```
Then we use the AI model to generate a time estimate for the task. We also wrapped this in a utility function, so the `addTodo` handler calls it:
```typescript theme={null}
const estimate = await aiEstimate(tenantNile, title.toString());
```
The `aiEstimate` function is defined in `lib/AiUtils.tsx` and first it looks up similar tasks using embeddings. Because we are still in the tenant
context (see the use of `tenantNile` client), we only look up tasks for this tenant:
```typescript theme={null}
const similarTasks = await tenantNile.query(
`SELECT title, estimate FROM todos WHERE embedding <-> $1 < 1`,
[embeddingToSQL(embedding)],
);
```
and finally, we use the AI model to generate the estimate. We include the similar tasks in the prompt to the model, so it can use them as reference:
```typescript theme={null}
const aiEstimate = await ai.chat.completions.create({
messages: [
{
role: 'user',
content: `you are an amazing project manager. I need to ${title}. How long do you think this will take?
I have a few similar tasks with their estimates, please use them as reference: ${similarTasks}.
respond with just the estimate, no yapping.`,
},
],
model: model,
});
```
This estimate is then stored in the database along with the task and its vector embedding.
# Build AI-Native B2B application with Postgres, NodeJS and React
Source: https://thenile.dev/docs/getting-started/languages/node
In this tutorial, you will learn to build a multi-tenant AI-native todo list application, using Nile with NodeJS, React and OpenAI's client.
We'll use Nile to provide us with virtual-tenant databases - isolating the tasks for each tenant, and we'll use the AI models to generate automated time estimates
for each task in the todo list. The estimates will be based on the task title, and estimates of similar tasks in the tenant's history.
This technique is known as [RAG architecture](https://www.thenile.dev/docs/ai-embeddings/rag).
1. Sign up for an invite to [Nile](https://thenile.dev) if you don't have one already
2. You should see a welcome message. Click on "Lets get started"
3. Give your workspace and database names, or you can accept the default auto-generated names. In order to complete this quickstart in a browser, make sure you select to "Use Token in Browser".
After you created a database, you will land in Nile's query editor.
For our todo list application, we'll need tables to store tenants, users and todos. Tenants and users already exists in Nile, they are built-in tables. We'll just need to create a table for todos.
```sql theme={null}
create table todos (
id uuid DEFAULT (gen_random_uuid()),
tenant_id uuid,
title varchar(256),
estimate varchar(256),
embedding vector(768),
complete boolean);
```
You will see the new table in the panel on the left side of the screen, and you can expand it to view the columns.
The embedding column is a vector representation of the task. When the user adds new tasks, we will use these embeddings to find
semantically related tasks and use this as a basis of our AI-driven time estimates. This technique - looking up related data using embeddings and
using this data with text generation models is a key part of [RAG (Retrieval Augumented Generation)](https://www.thenile.dev/docs/ai-embeddings/rag).
See the `tenant_id` column? By specifying this column, You are making the table **tenant aware**. The rows in it will belong to specific tenants. If you leave it out, the table is considered shared, more on this later.
In the left-hand menu, click on "Settings" and then select "Credentials".
Generate credentials and keep them somewhere safe. These give you access to
the database.
This example uses AI chat and embedding models to generate automated time
estimates for each task in the todo list. In order to use this functionality,
you will need access to models from a vendor with OpenAI compatible APIs. Make
sure you have an API key, API base URL and the [names of the models you'll
want to use](https://www.thenile.dev/docs/ai-embeddings/embedding_models).
Enough GUI for now. Let's get to some code.
If you haven't cloned this repository yet, now will be an excellent time to do so.
```bash theme={null}
git clone https://github.com/niledatabase/niledatabase
cd niledatabase/examples/quickstart/node_react
```
Rename `.env.example` to `.env`
Update NILE\_USER and NILE\_PASSWORD with the credentials you picked up in the previous step. It should look something like this:
```bash theme={null}
# Private env vars that should never show up in the browser
# These are used by the server to connect to Nile database
NILEDB_USER=018ad484-0d52-7274-8639-057814be60c3
NILEDB_PASSWORD=0d11b8e5-fbbc-4639-be44-8ab72947ec5b
# URL of the frontend, for the post-signup redirect
FE_URL = "http://localhost:3006"
NILEDB_API_URL=https://eu-central-1.api.dev.thenile.dev/databases/018ec979-2412-7062-9cda-35ae6fea7837
# for AI estimates
AI_API_KEY=your_api_key_for_openai_compatible_service
AI_BASE_URL=https://api.fireworks.ai/inference/v1
AI_MODEL=accounts/fireworks/models/llama-v3p1-405b-instruct
EMBEDDING_MODEL=nomic-ai/nomic-embed-text-v1.5
```
Install dependencies
```bash theme={null}
npm install
```
Start both NodeJS api server and the React frontend
```bash theme={null}
npm start
```
Go to [http://localhost:3000](http://localhost:3000) in a browser to see the app.
You can try a few things in the app:
* Sign up as a new user
* Create a tenant
* Create a todo task and see its time estimate. If you create more tasks, the estimates for new tasks will use the embeddings of the existing tasks to generate the estimates.
Go back to the Nile query editor and see the data you created from the app.
```sql theme={null}
SELECT tenants.name, title, estimate, complete
FROM todos join tenants on tenants.id = todos.tenant_id;
```
You should see all the todos you created, and the tenants they belong to.
This example is a good starting point for building your own application with Nile.
You have learned basic Nile concepts and how to use them with NodeJS and React.
You can learn more about Nile's tenant virtualization features in the following tutorials:
* [Tenant management](/tenant-virtualization/tenant-management)
* [Tenant isolation](/tenant-virtualization/tenant-isolation)
You can explore Nile's JS SDK in the [SDK reference](/auth/sdk-reference/javascript/overview).
You can learn [More about AI in Nile](https://www.thenile.dev/docs/ai-embeddings), or try a more advanced example like:
* [Chat with PDFs](https://www.thenile.dev/docs/getting-started/examples/chat_with_pdf)
* [Code Assistant](https://www.thenile.dev/docs/getting-started/examples/code_assistant)
## How does it work?
The interesting part of this example is the NodeJS server. Lets take a look at [`/examples/quickstart/node_react/src/be/app.ts`](https://github.com/niledatabase/niledatabase/blob/main/examples/quickstart/node_react/src/be/app.ts).
### Using AI models to generate estimates
This example uses AI chat and embedding models to generate automated time estimates for each task in the todo list. We handle the time estimates in
the route handler for `app.post("/api/tenants/:tenantId/todos"`. This handler executes when users add new tasks.
This is what the handler code looks like:
```javascript theme={null}
const similarTasks = await findSimilarTasks(nile, title);
const estimate = await aiEstimate(title, similarTasks);
const embedding = await embedTask(title, EmbeddingTasks.SEARCH_DOCUMENT);
const newTodo = await nile.query(
`INSERT INTO todos (tenant_id, title, complete, estimate, embedding)
VALUES ($1, $2, $3, $4, $5)
RETURNING *;`,
[
nile.tenantId, // setting from context
title,
complete || false,
estimate,
embeddingToSQL(embedding),
],
);
```
As you can see, we look up similar tasks and then use the AI model to generate the estimate. We then store the task, with the estimate and the task embedding in the database.
The stored embedding will be used to find similar tasks in the future. The methods `findSimilarTasks`, `aiEstimate` and `embedTask` are all defined in `AiUtils.ts`.
They are wrappers around standard AI model calls and database queries, and they handle the specifics of the AI model we are using.
This will make it easy to switch models in the future.
Getting similar tasks is done by querying the database for tasks with similar embeddings.
```javascript theme={null}
const embedding = await embedTask(title, EmbeddingTasks.SEARCH_QUERY);
// get similar tasks, no need to filter by tenant because we are already in the tenant context
const similarTasks = await nile.db.query(
`SELECT title, estimate FROM todos WHERE embedding <-> $1 < 1 ORDER BY embedding <-> $1 LIMIT 3;`,
[embeddingToSQL(embedding)],
);
```
We started by generating an embedding with `SEARCH_QUERY` task type. This is because we are looking for similar tasks to the new task. We use an embedding model
from the `nomic` family, which is trained to perform specific types of embedding tasks. Telling it that we are generating the embedding for a lookup vs
generating an embedding that we will store with the document (as we'll do in a bit), should help the model produce more relevant results.
As you can see, we filter out results where the cosine distance is higher than 1.
The lower the cosine distance is, the more similar the tasks are (0 indicate that they are identical).
A cosine distance of 1 means that the vectors are essentially unrelated, and when cosine distance is closer to 2, it indicates that the vectors are semantically opposites.
The `embedTask` function uses the embedding model to generate the embedding and is a very simple wrapper on the model:
```javascript theme={null}
let resp = await ai.embeddings.create({
model: EMBEDDING_MODEL,
input: adjust_input(title, task),
});
```
Now that we have the similar tasks, the handler calls `aiEstimate` to generate the time estimate.
This function also wraps a model, this time a conversation model rather than an embedding model. And it icludes the similar tasks in the promopt, so the model will
generate similar estimates:
```javascript theme={null}
const model =
process.env.AI_MODEL || 'accounts/fireworks/models/llama-v3p1-405b-instruct';
const aiEstimate = await ai.chat.completions.create({
messages: [
{
role: 'user',
content: `you are an amazing project manager. I need to ${title}. How long do you think this will take?
I have a few similar tasks with their estimates, please use them as reference: ${similarTasks}.
respond with just the estimate, keep the answer short.`,
},
],
max_tokens: 64, // limit the response to 64 tokens, to fit in our estimate field
model: model,
});
```
This estimate is then stored in the database along with the task and its vector embedding.
### Working with virtual tenant databases
The NodeJS server uses the [Nile JS client](https://github.com/niledatabase/nile-js) to connect to Nile.
When the Nile client is initialized, it uses the credentials you provided in the `.env` file to connect with the API:
```js theme={null}
const nile = Nile();
```
The application uses Express middleware to capture the tenant identity for the current request and set Nile context:
```js theme={null}
app.param('tenantId', (req, res, next, tenantId) => {
nile.setContext({ tenantId });
next();
});
```
We use Nile SDK to both execute SQL and make API calls to Nile. For example, to create as new tenant:
```js theme={null}
app.post("/api/tenants", async (req, res) => {
const { name } = req.body;
if (!name) {
res.status(400).json({
message: "No tenant name provided",
});
}
try {
const createTenantResponse = await nile.tenants.create({
name: name,
});
const tenant = await createTenantResponse.json();
res.json(tenant);
} catch (error: any) {
console.log("error creating tenant: " + error.message);
res.status(500).json({
message: "Internal Server Error",
});
}
});
```
# Build AI-Native B2B application with Postgres and Prisma
Source: https://thenile.dev/docs/getting-started/languages/prisma
In this tutorial, you will learn about Nile's tenant virtualization features, while building a todo list application.
We'll use Prisma as the ORM to interact with the database, OpenAI client to work with AI models, and Express as the web framework.
Nile will provide us with virtual-tenant databases - isolating the tasks for each tenant, and we'll use the AI models to generate automated time estimates
for each task in the todo list. The estimates will be based on the task title, and estimates of similar tasks in the tenant's history.
This technique is known as [RAG architecture](https://www.thenile.dev/docs/ai-embeddings/rag).
1. Sign up for an invite to [Nile](https://thenile.dev) if you don't have one already
2. You should see a welcome message. Click on "Lets get started"
3. Give your workspace and database names, or you can accept the default auto-generated names. In order to complete this quickstart in a browser, make sure you select to "Use Token in Browser".
After you created a database, you will land in Nile's query editor. Since our application requires a table for storing all the "todos" this is a good time to create one:
```sql theme={null}
CREATE TABLE IF NOT EXISTS "todos" (
"id" uuid DEFAULT gen_random_uuid(),
"tenant_id" uuid,
"title" varchar(256),
"estimate" varchar(256),
"embedding" vector(768),
"complete" boolean,
CONSTRAINT todos_tenant_id_id PRIMARY KEY("tenant_id","id")
);
```
You will see the new table in the panel on the left side of the screen, and you can expand it to view the columns.
The embedding column is a vector representation of the task. When the user adds new tasks, we will use these embeddings to find
semantically related tasks and use this as a basis of our AI-driven time estimates. This technique - looking up related data using embeddings and
using this data with text generation models is a key part of [RAG (Retrieval Augumented Generation)](https://www.thenile.dev/docs/ai-embeddings/rag).
See the `tenant_id` column? By specifying this column, You are making the table **tenant aware**. The rows in it will belong to specific tenants. If you leave it out, the table is considered shared, more on this later.
In the left-hand menu, click on "Settings" and then select "Connection".
Click on the Postgres button, then click "Generate Credentials" on the top right corner. Copy the connection string - it should now contain the credentials we just generated.
This example uses AI chat and embedding models to generate automated time
estimates for each task in the todo list. In order to use this functionality,
you will need access to models from a vendor with OpenAI compatible APIs. Make
sure you have an API key, API base URL and the [names of the models you'll
want to use](https://www.thenile.dev/docs/ai-embeddings/embedding_models).
Enough GUI for now. Let's get to some code.
If you haven't cloned this repository yet, now will be an excellent time to do so.
```bash theme={null}
git clone https://github.com/niledatabase/niledatabase
cd niledatabase/examples/quickstart/prisma
```
Rename `.env.example` to `.env`, and update it with the connection string you just copied from Nile Console and the configuration of your AI vendor and model.
Make sure you don't include the word "psql". It should look something like this:
```bash theme={null}
DATABASE_URL=postgres://018b778a-30df-7cdd-b55c-2f9664db39f3:ff3fb983-683c-4616-bbbc-519d8ddbbce5@db.thenile.dev:5432/gwen_db
# for AI estimates
AI_API_KEY=your_api_key_for_openai_compatible_service
AI_BASE_URL=https://api.fireworks.ai/inference/v1
AI_MODEL=accounts/fireworks/models/llama-v3p1-405b-instruct
EMBEDDING_MODEL=nomic-ai/nomic-embed-text-v1.5
```
Install dependencies with `yarn install` or `npm install`.
```bash theme={null}
npm install
```
Start the web service with `npm start` or `yarn start`.
Now you can use `curl` to explore the APIs. Here are a few examples:
```bash theme={null}
# create a tenant
curl --location --request POST 'localhost:3001/api/tenants' \
--header 'Content-Type: application/json' \
--data-raw '{"name":"my first customer", "id":"108124a5-2e34-418a-9735-b93082e9fbf2"}'
# get tenants
curl -X GET 'http://localhost:3001/api/tenants'
# create a todo (don't forget to use a read tenant-id in the URL)
curl -X POST \
'http://localhost:3001/api/tenants/108124a5-2e34-418a-9735-b93082e9fbf2/todos' \
--header 'Content-Type: application/json' \
--data-raw '{"title": "feed the cat", "complete": false}'
# list todos for tenant (don't forget to use a read tenant-id in the URL)
curl -X GET \
'http://localhost:3001/api/tenants/108124a5-2e34-418a-9735-b93082e9fbf2/todos'
# list todos for all tenants
curl -X GET \
'http://localhost:3001/insecure/all_todos'
```
Go back to the Nile query editor and see the data you created from the app.
```sql theme={null}
SELECT tenants.name, title, estimate, complete FROM todos join tenants
on tenants.id = todos.tenant_id;
```
You should see all the todos you created,
and the tenants they belong to.
This example is a good starting point for building your own application with Nile.
You have learned basic Nile, AI and RAG concepts and how to use them with Prisma.
You can learn more about this example in detail below or you can learn more about
Nile's tenant virtualization features in the following tutorials:
* [Tenant management](/tenant-virtualization/tenant-management)
* [Tenant isolation](/tenant-virtualization/tenant-isolation)
You can explore Nile's JS SDK in the [SDK reference](/auth/sdk-reference/javascript/overview).
You can learn [More about AI in Nile](https://www.thenile.dev/docs/ai-embeddings), or try a more advanced example like:
* [Chat with PDFs](https://www.thenile.dev/docs/getting-started/examples/chat_with_pdf)
* [Code Assistnat](https://www.thenile.dev/docs/getting-started/examples/code_assistant)
## How does it work?
Let's take a look at the code and understand how it works in detail.
### Working with Prisma data models
This example uses Prisma as the ORM. Prisma, like most ORMs, works by mapping the relational model in the database to an object model in the application.
When you cloned the example, you got a [Prisma schema file](https://github.com/niledatabase/niledatabase/blob/main/examples/quickstart/prisma/prisma/schema.prisma) that contains definitions for key tables (such as tenants and todos) and their relationships.
This file is used to generate the Prisma client, which is used by the application to interact with the database.
It can also be used to generate the database schema, but in this example we already created it in the Nile Console.
We generated the schema file from the database using the following commands:
```bash theme={null}
npx prisma init
npx prisma db pull
```
This is also known as Prisma's "introspection" feature. Because Prisma does not yet support [`pg_vector` extension](https://www.thenile.dev/docs/ai-embeddings/pg_vector)
and the vector data type, we had to manually add the vector column to the schema file:
```text theme={null}
model todos {
id String @default(dbgenerated("gen_random_uuid()")) @db.Uuid
tenant_id String @db.Uuid
title String? @db.VarChar(256)
estimate String? @db.VarChar(256)
// Prisma doesn't support vector types yet: https://github.com/prisma/prisma/issues/18442
embedding Unsupported("vector(768)")?
complete Boolean?
@@id([tenant_id, id], map: "todos_tenant_id_id")
@@schema("public")
}
```
We then generated a Prisma client with:
```bash theme={null}
npm install @prisma/client
npx prisma generate
```
### Querying tenant databases
Once we have a Prisma client, we can use it to run queries. For example, lets look at how we queried the database for a list of todos for a specific tenant:
```typescript theme={null}
const tenantDB = tenantContext.getStore();
// No need for a "where" clause here because we are setting the tenant ID in the context
const todos = await tenantDB?.todos.findMany();
res.json(todos);
```
The query looks like a regular Prisma query, but it's actually running against a tenant database, which is why it doesn't need to filter the todos.
As you can see in the snippet above, we are getting the tenant database from `tenantContext`, lets take a look at what is this context and how it is managed.
We also have queries that use the `embedding` column with the `vector` data type. This column is used to store embeddings for each task, which are used to generate automated time estimates for each task in the todo list.
Because Prisma does not yet support the `vector` data type, we need to use Prisma's `executeRaw` and `queryRaw` methods to work with this column.
We'll show the example in the next section when we talk about the use of AI models to generate time estimates.
After this small detour, we'll get back to the tenant context and how to use Prisma with virtual tenant databases.
### Using AI models to estimate time to complete tasks
This example uses AI chat and embedding models to generate automated time estimates for each task in the todo list. We handle the time estimates in
the route handler for `app.post("/api/tenants/:tenantId/todos"`. This handler executes when users add new tasks.
This is what the handler code looks like:
```typescript theme={null}
const tenantDB = tenantContext.getStore();
// ... some setup redacted...
// We are using tenantDB with tenant context to ensure that we only find tasks for the current tenant
const similarTasks = await findSimilarTasks(tenantDB, title);
console.log('found similar tasks: ' + JSON.stringify(similarTasks));
const estimate = await aiEstimate(title, similarTasks);
console.log('estimated time: ' + estimate);
// get the embedding for the task, so we can find it in future similarity searches
const embedding = await embedTask(title, EmbeddingTasks.SEARCH_DOCUMENT);
console.log('tenant_id: ' + tenantId);
// This is safe because Nile validates the tenant ID and protects against SQL injection
const newTodo = await tenantDB.$queryRawUnsafe(
`INSERT INTO todos (tenant_id, title, complete, estimate, embedding) VALUES ('${tenantId}', $1, $2, $3, $4::vector)
RETURNING id, title, complete, estimate`,
title,
complete,
estimate,
embeddingToSQL(embedding),
);
res.json(newTodo);
```
As you can see, we look up similar tasks and then use the AI model to generate the estimate. We then store the task, with the estimate and the task embedding in the database.
The stored embedding will be used to find similar tasks in the future. The methods `findSimilarTasks`, `aiEstimate` and `embedTask` are all defined in `AiUtils.ts`.
They are wrappers around standard AI model calls and database queries, and they handle the specifics of the AI model we are using.
This will make it easy to switch models in the future.
Note that when we save the todo with the estimate and embedding, we use `tenantDB.$queryRawUnsafe` to insert the data.
This is because Prisma does not yet support the `vector` data type, so we need to go a bit lower level.
Getting similar tasks is done by querying the database for tasks with similar embeddings:
```typescript theme={null}
const embedding = await embedTask(title, EmbeddingTasks.SEARCH_QUERY);
// get similar tasks, no need to filter by tenant because we are already in the tenant context
const similarTasks =
await tenantNile.$queryRaw`SELECT title, estimate FROM todos WHERE
embedding <-> ${embeddingToSQL(
embedding,
)}::vector < 1 order by embedding <-> ${embeddingToSQL(
embedding,
)}::vector limit 3`;
console.log(` found ${similarTasks.length} similar tasks`);
return similarTasks;
```
We started by generating an embedding with `SEARCH_QUERY` task type. This is because we are looking for
similar tasks to the new task. We use an embedding model from the `nomic` family, which is trained to perform specific types of embedding tasks. Telling it that we are generating the embedding for a lookup vs
generating an embedding that we will store with the document (as we'll do in a bit), should help the model produce more relevant results.
We use `queryRaw` to run the query because Prisma does not yet support the `vector` data type, and we need to use both the
embedding column, and the `<->` cosine distance operator in order to find the most similar tasks.
As you can see, we filter out results where the cosine distance is higher than 1.
The lower the cosine distance is, the more similar the tasks are (0 indicate that they are identical).
A cosine distance of 1 means that the vectors are essentially unrelated, and when cosine distance is closer to 2, it indicates that the vectors are semantically opposites.
The `embedTask` function uses the embedding model to generate the embedding and is a very simple wrapper on the model:
```typescript theme={null}
let resp = await ai.embeddings.create({
model: EMBEDDING_MODEL,
input: adjust_input(title, task),
});
```
Now that we have the similar tasks, the handler calls `aiEstimate` to generate the time estimate.
This function also wraps a model, this time a conversation model rather than an embedding model. And it icludes the similar tasks in the promopt, so the model will
generate similar estimates:
```typescript theme={null}
const aiEstimate = await ai.chat.completions.create({
messages: [
{
role: 'user',
content: `you are an amazing project manager. I need to ${title}. How long do you think this will take?
I have a few similar tasks with their estimates, please use them as reference: ${similarTasks}.
respond with just the estimate, keep the answer short.`,
},
],
max_tokens: 64, // limit the response to 64 tokens
model: model,
});
```
As you've seen at the start of this section when we reviewed the route handler for `app.post("/api/tenants/:tenantId/todos"`, this estimate is then stored in the database along with the task and its vector embedding.
### Tenant context
Tenant context uses `AsyncLocalStorage` to store a Prisma Client with additional tenant information.
`AsyncLocalStorage` is a Node.js feature that allows you to store data in a context that is local to the current execution flow.
```typescript theme={null}
const tenantContext = new AsyncLocalStorage();
```
We use Express middleware to get the tenant ID from the HTTP request, use it to initialize a Prisma client and set it in the context:
```typescript theme={null}
app.use((req, res, next) => {
const fn = match('/api/tenants/:tenantId/todos', {
decode: decodeURIComponent,
});
const m = fn(req.path);
const tenantId = m?.params?.tenantId;
console.log('setting context to tenant: ' + tenantId);
tenantContext.run(
prisma.$extends(tenantDB(tenantId)) as any as PrismaClient,
next,
);
});
```
`tenantContext.run()` stores the Prisma Client in a local storage, accessible from anywhere in the current request execution flow. The second argument is a callback function that will be executed after the context is set.
Since this is an Express middleware, the callback is the next middleware or request handler in the chain.
The only last bit to understand is the initialization of the Prisma Client with `tenantDB`.
### Initializing the Prisma Client with tenant information
Lets take a look at the tenantDB object:
```typescript theme={null}
function tenantDB(
tenantId: string | null | undefined,
): (client: any) => PrismaClient {
return Prisma.defineExtension((prisma) =>
// @ts-ignore (Excessive stack depth comparing types...)
prisma.$extends({
query: {
$allModels: {
async $allOperations({ args, query }) {
// set tenant context, if tenantId is provided
// otherwise, reset it
const [, result] = tenantId
? await prisma.$transaction([
prisma.$executeRawUnsafe(
`SET nile.tenant_id = '${tenantId}';`,
),
query(args),
])
: await prisma.$transaction([
prisma.$executeRawUnsafe(`RESET nile.tenant_id;`),
query(args),
]);
return result;
},
},
},
}),
);
}
```
While it looks a bit mysterious, it's actually quite simple. This function takes a tenant ID and returns a Prisma extension.
The extension is a function that takes the Prisma Client as an argument and returns a new Prisma Client with additional functionality.
In this case, the additional functionality is to run every operation, for every model in the database, in a transaction that sets the tenant ID in the database session.
### Tying it all together
Lets circle back to what happens when we call:
```bash theme={null}
curl -X GET \
'http://localhost:3001/api/tenants/108124a5-2e34-418a-9735-b93082e9fbf2/todos'
```
First, the Express middleware extracts the tenant ID from the URL and creates a Prisma Client with an extension that will set the tenant ID in the database session before each operation.
It then uses `AsyncLocalStorage` to store the Prisma Client in a context that is local to the current request execution flow.
The request handler then uses the Prisma Client from the context to run a query against the database. Since the tenant ID is already set in the database session, the query doesn't need to filter the todos by tenant ID.
```typescript theme={null}
const tenantDB = tenantContext.getStore();
// No need for a "where" clause here because we are setting the tenant ID in the context
const todos = await tenantDB?.todos.findMany();
res.json(todos);
```
Other request handlers in the application use the same pattern to insert or update new todos for each tenant.
# Build AI-Native B2B application with Postgres and Python
Source: https://thenile.dev/docs/getting-started/languages/python
In this tutorial, you will learn to build a multi-tenant AI-native todo list application, using Nile with Python, FastAPI, SQLAlchemy, and OpenAI's client.
We'll use Nile to provide us with virtual-tenant databases - isolating the tasks for each tenant, and we'll use the AI models to generate automated time estimates
for each task in the todo list.
1. Sign up for an invite to [Nile](https://thenile.dev) if you don't have one already
2. You should see a welcome message. Click on "Lets get started"
3. Give your workspace and database names, or you can accept the default auto-generated names. In order to complete this quickstart in a browser, make sure you select to "Use Token in Browser".
After you created a database, you will land in Nile's query editor.
For our todo list application, we'll need tables to store tenants, users and todos.
Tenants and users already exists in Nile, they are built-in tables and you can see them in the list on the left side of the screen.
We'll just need to create a table for todos.
```sql theme={null}
create table todos (
id uuid DEFAULT (gen_random_uuid()),
tenant_id uuid,
title varchar(256),
estimate varchar(256),
embedding vector(768),
complete boolean);
```
You will see the new table in the panel on the left side of the screen, and you can expand it to view the columns.
The embedding column is a vector representation of the task. When the user adds new tasks, we will use these embeddings to find
semantically related tasks and use this as a basis of our AI-driven time estimates. This technique - looking up related data using embeddings and
using this data with text generation models is called [RAG (Retrieval Augumented Generation)](https://www.thenile.dev/docs/ai-embeddings/rag).
See the `tenant_id` column? By specifying this column, You are making the table **tenant aware**. The rows in it will belong to specific tenants. If you leave it out, the table is considered shared, more on this later.
In the left-hand menu, click on "Settings" and then select "Credentials".
Generate credentials and keep them somewhere safe. These give you access to
the database.
This example uses AI chat and embedding models to generate automated time
estimates for each task in the todo list. In order to use this functionality,
you will need access to models from a vendor with OpenAI compatible APIs. Make
sure you have an API key, API base URL and the [names of the models you'll
want to use](https://www.thenile.dev/docs/ai-embeddings/embedding_models).
Enough GUI for now. Let's get to some code.
If you haven't cloned this repository yet, now will be an excellent time to do so.
```bash theme={null}
git clone https://github.com/niledatabase/niledatabase
cd niledatabase/examples/quickstart/python
```
Copy `.env.example` to `.env` and fill in the details of your Nile DB. The ones you copied and kept safe in step 3.
It should look something like this:
```bash theme={null}
DATABASE_URL=postgresql://user:password@db.thenile.dev:5432/mydb
LOG_LEVEL=DEBUG
SECRET_KEY = "09d25e094faa6ca2556c818166b7a9563b93f7099f6f0f4caa6cf63b88e8d3e7"
ALGORITHM = "HS256"
ACCESS_TOKEN_EXPIRE_MINUTES = 30
# for AI estimates
AI_API_KEY=your_api_key_for_openai_compatible_service
AI_BASE_URL=https://api.fireworks.ai/inference/v1
AI_MODEL=accounts/fireworks/models/llama-v3p1-405b-instruct
EMBEDDING_MODEL=nomic-ai/nomic-embed-text-v1.5
```
Optional, but recommended, step is to set up a virtual Python environment:
```bash theme={null}
python -m venv venv
source venv/bin/activate
```
Then, install dependencies:
```bash theme={null}
pip install -r requirements.txt
```
If you'd like to use the app with the UI, you'll want to build the UI assets first:
```bash theme={null}
cd ui
npm install
npm run build
```
Then start the Python webapp:
```bash theme={null}
uvicorn main:app --reload
```
Go to [http://localhost:8000](http://localhost:8000) in a browser to see the app.
You can try a few things in the app:
* Sign up as a new user
* Create a tenant
* Create a todo task and see its time estimate. If you create more tasks, the estimates for new tasks will use the embeddings of the existing tasks to generate the estimates.
You can also use the API directly:
```bash theme={null}
# login
curl -c cookies -X POST 'http://localhost:8000/api/login' \
--header 'Content-Type: application/json' \
--data-raw '{"email":"test9@pytest.org","password":"foobar"}'
# create tenant
curl -b cookies -X POST 'localhost:8000/api/tenants' \
--header 'Content-Type: application/json' \
--data-raw '{"name":"my first customer"}'
# list tenants
curl -b cookies -X GET 'http://localhost:8000/api/tenants'
# create todo for a tenant (make sure you replace the tenant ID with the one you got from the previous step)
curl -b cookies -X POST \
'http://localhost:8000/api/todos' \
--header 'Content-Type: application/json' \
--header 'X-Tenant-Id: 3c9bfcd0-7702-4e0e-b3f0-4e84221e20a7' \
--data-raw '{"title": "feed the cat", "complete": false}'
# list todos for a tenant (make sure you replace the tenant ID with the one you got from the previous step)
curl -b cookies -X GET \
--header 'X-Tenant-Id: 3c9bfcd0-7702-4e0e-b3f0-4e84221e20a7' \
'http://localhost:8000/api/todos'
```
Go back to the Nile query editor and see the data you created from the app.
```sql theme={null}
SELECT tenants.name, title, estimate, complete
FROM todos join tenants on tenants.id = todos.tenant_id;
```
You should see all the todos you created, and the tenants they belong to.
This example is a good starting point for building your own application with Nile.
You have learned basic Nile concepts and how to use them with Python, FastAPI, and SQLAlchemy.
You can learn more about Nile's tenant virtualization features in the following tutorials:
* [Tenant management](/tenant-virtualization/tenant-management)
* [Tenant isolation](/tenant-virtualization/tenant-isolation)
You can learn [More about AI in Nile](https://www.thenile.dev/docs/ai-embeddings), or try a more advanced example like:
* [Chat with PDFs](https://www.thenile.dev/docs/getting-started/examples/chat_with_pdf)
* [Code Assistant](https://www.thenile.dev/docs/getting-started/examples/code_assistant)
## How does it work?
The app uses FastAPI, a modern Python web framework, and SQLAlchemy, a popular ORM. The app is built with tenants in mind, and it uses Nile's tenant context to isolate data between tenants.
`main.py` is the entry point of the app. It sets up the FastAPI app, registers the middleware and has all the routes.
### Using AI models for time estimates
This example uses AI chat and embedding models to generate automated time estimates for each task in the todo list. We handle the time estimates in
the `create_todo` method which is the route handler for `@app.post("/api/todos")`. This handler executes when users add new tasks.
This is what the handler code looks like:
```python theme={null}
similar_tasks = get_similar_tasks(session, todo.title)
logger.info(f"Generating estimate based on similar tasks: {similar_tasks}")
estimate = ai_estimate(todo.title, similar_tasks)
embedding = get_embedding(todo.title, EmbeddingTasks.SEARCH_DOCUMENT)
todo.embedding = embedding
todo.estimate = estimate
session.add(todo)
session.commit()
```
As you can see, we look up similar tasks and then use the AI model to generate the estimate. We then store the task, with the estimate and the task embedding in the database.
The stored embedding will be used to find similar tasks in the future. The methods `get_similar_tasks`, `ai_estimate` and `get_embedding` are all defined in `ai_utils.py`.
They are wrappers around standard AI model calls and database queries, and they handle the specifics of the AI model we are using.
This will make it easy to switch models in the future.
Getting similar tasks is done by querying the database for tasks with similar embeddings. Before we search the database, we need to generate the embedding for the new task:
```python theme={null}
def get_similar_tasks(session: any, text: str):
query_embedding = get_embedding(text, EmbeddingTasks.SEARCH_QUERY)
similar_tasks_raw = (
session.query(Todo)
.filter(Todo.embedding.cosine_distance(query_embedding) < 1)
.order_by(Todo.embedding.cosine_distance(query_embedding)).limit(3))
return [{"title": task.title, "estimate": task.estimate} for task in similar_tasks_raw]
```
We started by generating an embedding with `SEARCH_QUERY` task type. This is because we are looking for similar tasks to the new task. We use an embedding model
from the `nomic` family, which is trained to perform specific types of embedding tasks. Telling it that we are generating the embedding for a lookup vs
generating an embedding that we will store with the document (as we'll do in a bit), should help the model produce more relevant results.
In order to use vector embeddings with SQL Alchemy and SQL Model ORM, we used [PG Vector's Python library](https://github.com/pgvector/pgvector-python).
You'll find it in `requirements.txt` for the project. Note that we filter out results where the cosine distance is higher than 1.
The lower the cosine distance is, the more similar the tasks are (0 indicate that they are identical).
A cosine distance of 1 means that the vectors are essentially unrelated, and when cosine distance is closer to 2, it indicates that the vectors are semantically opposites.
The `get_embedding` function uses the embedding model to generate the embedding and is a very simple wrapper on the model:
```python theme={null}
response = client.embeddings.create(
model=os.getenv("EMBEDDING_MODEL"),
input=adjust_input(text, task),
)
```
Now that we have the similar tasks, the handler calls `ai_estimate` to generate the time estimate.
This function also wraps a model, this time a conversation model rather than an embedding model. And it icludes the similar tasks in the promopt, so the model will
generate similar estimates:
```python theme={null}
response = client.chat.completions.create(
model = os.getenv("AI_MODEL"),
messages = [
{
"role": "user",
"content" :
f'you are an amazing project manager. I need to {text}. How long do you think this will take?'
f'I have a few similar tasks with their estimates, please use them as reference: {similar_tasks}.'
f'respond with just the estimate, no yapping.',
},
],
)
```
This estimate is then stored in the database along with the task and its vector embedding.
### Working with virtual tenant databases
The first thing we do in the app is set up the tenant middleware. The `TenantAwareMiddleware` is defined in `middleware.py`,
and it is responsible for extracting the tenant ID from the request headers and setting it in the database session.
This ensures that all database operations are performed in the context of the current tenant.
```python theme={null}
app = FastAPI()
app.add_middleware(TenantAwareMiddleware)
```
The middleware runs before any request is processed. But not every request has a tenant context. For example, `login` or `create_tenant` routes doesn't need a tenant context.
Requests that don't have a tenant context are considered to be `global` since they are performed on the database as a whole, not in the virtual database for a specific tenant.
To handle a request in the global context, we use a global session. This is a session that doesn't have a tenant context. For example to create a new tenant:
```python theme={null}
@app.post("/api/tenants")
async def create_tenant(tenant:Tenant, request: Request, session = Depends(get_global_session)):
session.add(tenant)
session.commit()
return tenant
```
To handle a request in the tenant context, we use a tenant session. This is a session that has a tenant context. For example to list todos:
```python theme={null}
@app.get("/api/todos")
async def get_todos(session = Depends(get_tenant_session)):
todos = session.query(Todo).all()
return todos
```
This looks like it could return all todos from all tenants, but it doesn't. The `get_tenant_session`
function sets the tenant context for the session, and the query is executed in the virtual database of the tenant.
The last piece of the puzzle is the `get_tenant_session` function. It is defined in `db.py` and is responsible for creating the session with the correct context.
```python theme={null}
def get_tenant_session():
session = Session(bind=engine, expire_on_commit=False)
try:
tenant_id = get_tenant_id()
user_id = get_user_id()
session.execute(text(f"SET nile.tenant_id='{tenant_id}';"))
session.execute(text(f"SET nile.user_id='{user_id}';"))
yield session
except:
session.rollback()
raise
finally:
session.execute(text("RESET nile.user_id;"))
session.execute(text("RESET nile.tenant_id;"))
session.commit()
pass
```
We are setting both the user and tenant context in the session. This is important for security and isolation.
The user context is used to check if the user has access to the tenant, and the tenant context is used to isolate the data.
Note that we are using FastAPI dependency injection to get the session in the route handlers. This is a powerful feature of FastAPI that makes it easy to manage resources like sessions.
The `yield` keyword is used to return the session to the caller, and the `finally` block is used to clean up the session after the request is processed.
And this is it. Thats all we need to do to build a multi-tenant app with Nile, FastAPI and SQLAlchemy.
## 8. Looking good!
🏆 Tada! You have learned the basic Nile concepts:
* [Tenant management](/tenant-virtualization/tenant-management)
* [Tenant isolation](/tenant-virtualization/tenant-isolation)
# Explore Nile in 5 minutes with PSQL
Source: https://thenile.dev/docs/getting-started/languages/sql
Learn about Nile's tenant aware tables and how they provide tenant isolation through virtual tenant databases.
1. Sign up for an invite to [Nile](https://thenile.dev) if you don't have one already
2. You should see a welcome message. Click on "Lets get started"
3. Give your workspace and database names, or you can accept the default auto-generated names. In order to complete this quickstart in a browser, make sure you select to "Use Token in Browser".
Lets imagine that we are building a todo list app that maintains task for each tenant/customer and also uses AI to predict the time required to complete the task.
Let us create our first table that has a tenant\_id column and a vector:
```sql theme={null}
CREATE TABLE IF NOT EXISTS "todos" (
"id" uuid DEFAULT gen_random_uuid(),
"tenant_id" uuid,
"title" varchar(256),
"estimate" varchar(256),
"embedding" vector(3),
"complete" boolean,
CONSTRAINT todos_pkey PRIMARY KEY("tenant_id","id")
);
```
You will see the new table in the panel on the left side of the screen, and you can expand it to view the columns. If you are in `psql`, you can do `\d todos` in order to view the schema.
See the `tenant_id` column? By specifying this column, You are making the table **tenant aware**. The rows in it will belong to specific tenants. If you leave it out, the table is considered shared, more on this later.
Nile ships with built-in tables, like `tenants` table that you used earlier. They are covered in more depth in our concepts documentation.
Meanwhile, lets insert a tenant into the built-in tenant table:
```sql theme={null}
-- Creating the first tenant
insert into tenants (id, name) values ('d24419bf-6122-4879-afa5-1d9c1b051d72', 'my first customer');
select * from tenants;
```
Now, with the new tenant created, lets insert a record to the "todos" table. For the first task, we simply retrieve the embedding and the estimate from the large language model using the title of the task.
Let us assume we got the embedding and the estimate from the model as \[1,2,3] and 1h
```sql theme={null}
-- adding a todo item for this tenant.
insert into todos (tenant_id, title, estimate, embedding, complete) values ('d24419bf-6122-4879-afa5-1d9c1b051d72', 'feed my cat', '1h', '[1,2,3]', false);
SELECT tenants.name, title, embedding, estimate, complete
FROM todos join tenants on tenants.id = todos.tenant_id;
```
If all went well, you'll see the first todo of your first customer 🏆
```
name title embedding estimate complete
-------------------------------------------------------------------------------
my first customer feed my cat [1,2,3] 1h false
```
It is now time for our second customer and their todo item:
```sql theme={null}
-- creating my second tenant
insert into tenants (id, name) values ('7e93c45f-fe65-4f26-8ab6-922850fa4c7a', 'second customer');
select * from tenants;
-- a new todo item for our second tenant
insert into todos (tenant_id, title, estimate, embedding, complete) values ('7e93c45f-fe65-4f26-8ab6-922850fa4c7a', 'take out the trash', '2h', '[0.8,0.2,0.6]', false);
SELECT tenants.name, title, embedding, estimate, complete
FROM todos join tenants on tenants.id = todos.tenant_id;
```
If all went well, you now see the todo items for both customers:
```
name title embedding estimate complete
-------------------------------------------------------------------------------------------
my first customer feed my cat [1,2,3] 1h false
second customer take out the trash [0.8,0.2,0.6] 2h false
```
Nile goes a step further and provides tenant isolation. You can set the session to a specific tenant, and every query that follows will only return data that belongs to this specific tenant.
Think of it as querying a virtual database dedicated to a tenant.
You can select a tenant either from the drop-down list next to the ▶️ button. Or by setting the session parameter in SQL.
```sql theme={null}
-- set context to isolate query to a specific tenant DB
-- our example uses the second tenant here
set nile.tenant_id = '7e93c45f-fe65-4f26-8ab6-922850fa4c7a';
SELECT tenants.name, title, embedding, estimate, complete
FROM todos join tenants on tenants.id = todos.tenant_id;
```
If all went well, you'll see the todo task for second customer" and not the first customer:
```
name title embedding estimate complete
-------------------------------------------------------------------------------------
second customer take out the trash [0.8,0.2,0.6] 2h false
```
The embedding for the new todo task is calculated first from the LLM (let's say this is \[0.78,0.18,0.62]). We will get similar tasks for the second customer based on the new task's embedding.
Note that we don't need to specify the second customer in the query since the session is already pointing to the second customer's virtual DB.
```sql theme={null}
-- Note the tenant context is already set to the second customer and the session points to that tenant's virtual DB
SELECT title, estimate FROM todos WHERE embedding <-> '[0.78,0.18,0.62]' < 1;
```
```
title estimate
------------------------------
take out the trash 2h
```
We can now ask the LLM to estimate the time for the new task by providing the estimates for similar tasks for the second customer.
Assume the LLM returned 1.5h.
```sql theme={null}
-- a new todo item for our second tenant
insert into todos (tenant_id, title, estimate, embedding, complete) values ('7e93c45f-fe65-4f26-8ab6-922850fa4c7a', 'clean the house', '1.5h', '[0.78,0.18,0.62]', false);
SELECT tenants.name, title, embedding, estimate, complete
FROM todos join tenants on tenants.id = todos.tenant_id;
```
Note that you only see the two tasks for the second customer. This is because the session is still pointing to the second customer's virtual DB.
```
name title embedding estimate complete
----------------------------------------------------------------------------------------------
second customer take out the trash [0.8,0.2,0.6] 2h false
second customer clean the house [0.78,0.18,0.62] 2h false
```
🏆 Tada! You have learned the key Nile concepts. And it only took 5 minutes.
You can learn more about Nile's tenant virtualization features in the following tutorials:
* [Tenant management](/tenant-virtualization/tenant-management)
* [Tenant isolation](/tenant-virtualization/tenant-isolation)
Next, you will probably want to learn how to use Nile for building an app in your favorite language.
Check out our [Getting Started](/getting-started/languages/python) guides for more information.
# Getting started with Nile's Postgres platform locally with Docker
Source: https://thenile.dev/docs/getting-started/postgres_docker
Nile provides a cloud offering to help build multi-tenant apps. You can also get started with Nile's Docker image
and try Nile locally. [Join our discord](https://discord.com/invite/8UuBB84tTy) to give feedback or ask questions about running Nile locally.
* [Docker](https://www.docker.com/get-started)
* Postgres client. We'll use `psql` in this guide.
```bash theme={null}
docker run -p 5432:5432 -p 3000:3000 -ti ghcr.io/niledatabase/testingcontainer:latest
```
This will start a Postgres database with Nile extensions installed. It will also start Nile Auth (optional).
If this is the first time you are running the container, it will also pull the latest image,create the `test` database
and the `00000000-0000-0000-0000-000000000000` user.
You can use `psql` with the following connection string:
```bash theme={null}
psql postgres://00000000-0000-0000-0000-000000000000:password@localhost:5432/test
```
Or, if you are using a different client, you use the following connection details:
```
Host: localhost
Port: 5432
Database: test
Username: 00000000-0000-0000-0000-000000000000
Password: password
```
From this point, you can use the local database just like you would use Nile service.
All the examples in the documentation are also applicable to the local database.
Below we'll go through the steps in the quickstart guide using the local database.
[Tenant-aware tables](/tenant-virtualization/tenant-isolation) are tables that have
a `tenant_id` column. All the rows in such tables belong to a specific tenant.
Let us create our first table that has a tenant\_id column and a vector column:
```sql theme={null}
CREATE TABLE IF NOT EXISTS "todos" (
"id" uuid DEFAULT gen_random_uuid(),
"tenant_id" uuid,
"title" varchar(256),
"estimate" varchar(256),
"embedding" vector(3),
"complete" boolean,
CONSTRAINT todos_pkey PRIMARY KEY("tenant_id","id")
);
```
If you are using `psql`, you can view the table schema by running `\d todos`.
Nile ships with built-in tables, like `tenants` table. Lets create our first tenant by inserting a row into the `tenants` table:
```sql theme={null}
-- Creating the first tenant
insert into tenants (id, name)
values ('d24419bf-6122-4879-afa5-1d9c1b051d72', 'my first customer');
select * from tenants;
```
Now that we have a tenant, we can insert data into our tenant-aware table:
```sql theme={null}
-- adding a todo item for this tenant
insert into todos (tenant_id, title, estimate, embedding, complete)
values ('d24419bf-6122-4879-afa5-1d9c1b051d72', 'feed my cat', '1h', '[1,2,3]', false);
```
and you can verify the data was inserted correctly by running:
```sql theme={null}
select * from todos;
```
You can add another tenant and insert data for that tenant in a similar fashion. This will allow us to explore
tenant isolation (in the next section).
```sql theme={null}
-- creating my second tenant
insert into tenants (id, name)
values ('7e93c45f-fe65-4f26-8ab6-922850fa4c7a', 'second customer');
select * from tenants;
insert into todos (tenant_id, title, estimate, embedding, complete)
values ('7e93c45f-fe65-4f26-8ab6-922850fa4c7a', 'take out the trash', '2h', '[0.8,0.2,0.6]', false);
select * from todos;
```
Nile goes a step further and provides tenant isolation. You can set the session to a specific tenant, and every query that follows will only return data that belongs to this specific tenant.
Think of it as querying a virtual database dedicated to this one specific tenant.
```sql theme={null}
-- set context to isolate query to a specific tenant DB
-- our example uses the second tenant here
set nile.tenant_id = '7e93c45f-fe65-4f26-8ab6-922850fa4c7a';
SELECT tenants.name, title, embedding, estimate, complete
FROM todos join tenants on tenants.id = todos.tenant_id;
```
✏️ Note that the container uses ephemeral storage, so all the data will be lost when the container is stopped.
This is intentional, as it simplifies the setup (and more importantly - the cleanup), while still allowing you to experiment and test your application.
### Looking good! What's next?
🏆 Tada! You have learned the key Nile concepts. And it only took 5 minutes.
You can learn more about Nile's tenant virtualization features in the following tutorials:
* [Tenant management](/tenant-virtualization/tenant-management)
* [Tenant isolation](/tenant-virtualization/tenant-isolation)
Next, you will probably want to learn how to use Nile for building an app in your favorite language.
Check out our [Getting Started](/getting-started/languages/sql) guides for more information.
## Optional: Docker container configuration
The docker container can be configured with the following environment variables:
* `NILE_TESTING_DB_NAME`: The name of the database. Defaults to `test`.
* `NILE_TESTING_DB_ID`: The ID of the database. Must be a UUID. Defaults to `01000000-0000-7000-8000-000000000000`.
* `NILE_TESTING_DB_USER`: The username for the database user. Must be UUID. Defaults to `00000000-0000-0000-0000-000000000000`.
* `NILE_TESTING_DB_PASSWORD`: The password of the database user. Defaults to `password`.
If you need to change the default values, you can do so by setting the environment variables when running the container.
```bash theme={null}
docker run -p 5432:5432 -ti \
-e NILE_TESTING_DB_NAME=mydatabase \
-e NILE_TESTING_DB_PASSWORD=mypassword \
ghcr.io/niledatabase/testingcontainer:v0.0.2
```
with this configuration, the connection string will be:
```bash theme={null}
psql postgres://00000000-0000-0000-0000-000000000000:mypassword@127.0.0.1:5432/mydatabase
```
You can also change the port mappings in the docker run command, if you want Postgres to listen on a different port.
## Troubleshooting
If you are having trouble running the container, the first step is to check that you are using the latest version of the container.
You can do this by first deleting existing images of `niledatabase/testingcontainer:latest` (since they may refer to older versions) and then running `docker pull ghcr.io/niledatabase/testingcontainer:latest`.
Please check if any of the common issues and solutions below help solve your problem. If not, we welcome you to report the issue on either [GitHub](https://github.com/niledatabase/niledatabase) or [Discord](https://discord.com/invite/8UuBB84tTy)
When reporting the issue, please include:
* The docker container logs (`docker logs `)
* The exact error message you're encountering
Here are some common issues you might encounter when running the docker container:
### Server closed the connection unexpectedly
It takes a bit of time for the container to start up. If you get the following error:
```bash theme={null}
➜ psql postgres://00000000-0000-0000-0000-000000000000:password@localhost:5432/test
psql: error: connection to server at "localhost" (::1), port 5432 failed: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
```
Wait for a few seconds, until the container logs indicate that the database is ready:
```bash theme={null}
Database has been created and is ready
```
Then try connecting again.
### Role does not exist
If you get the following error:
```bash theme={null}
➜ niledatabase_private git:(extension_groups) ✗ psql postgres://00000000-0000-0000-0000-000000000000:password@localhost:5432/test
psql: error: connection to server at "localhost" (::1), port 5432 failed: FATAL: role "00000000-0000-0000-0000-000000000000" does not exist
```
It is likely that you have another Postgres instance running on your machine that also uses port 5432. In this situation, `psql` will try to connect to the default Postgres instance, which fails because we are using a specific user that doesn't exist in the default Postgres instance.
You can either:
* Stop the other Postgres instance - the exact command depends on your operating system and Postgres installation method.
* Change the port mapping when running the container. For example to port 5433, and then connect to the new port:
```bash theme={null}
docker run -p 5433:5432 -ti ghcr.io/niledatabase/testingcontainer:latest
psql postgres://00000000-0000-0000-0000-000000000000:password@localhost:5433/test
```
# Integration testing with Nile's Postgres platform using Docker
Source: https://thenile.dev/docs/getting-started/postgres_testing
Nile provides a cloud offering to help build multi-tenant apps. However, it is a lot easier to build and iterate
locally before using the cloud solution. Nile Docker provides all the tools required to build and test locally before
moving to the cloud solution.
The image is running all the extensions and components that Nile uses as part of our cloud offering except our global
control plane used to manage our customers and billing.
The Docker image includes:
* 3 Postgres instances, to emulate a distributed production-like setup
* Each Postgres instance has all the Nile extensions installed
* All functionalities documented in our docs work with the image
* At this point, the UI console is not included in the image
[Join our discord](https://discord.com/invite/8UuBB84tTy) to give feedback or ask questions about running and testing with Nile's Docker image.
## Using Nile's docker container with TestContainers
A common way to automate integration tests is using [TestContainers](https://testcontainers.org/).
### Node.js
The following example shows how to use Nile's docker container with Node.js and
the [TestContainers](https://testcontainers.org/) library. It exposes a Postgres port and uses
`waitForLog` strategy to wait until the container is ready. Then it connects to the database and runs a simple query.
Make sure you install `testcontainers` and `pg` npm packages:
```bash theme={null}
npm install testcontainers pg
```
and then you can run the following snippet:
```typescript theme={null}
import { GenericContainer, Wait } from 'testcontainers';
import pg from 'pg';
const image = 'ghcr.io/niledatabase/testingcontainer:latest';
const container = await new GenericContainer(image)
.withExposedPorts(5432)
.withWaitStrategy(
Wait.forLogMessage('Database has been created and is ready'),
)
.start();
const client = new pg.Client({
host: container.getHost(),
port: container.getMappedPort(5432),
user: '00000000-0000-0000-0000-000000000000',
password: 'password',
database: 'test',
});
await client.connect();
const res = await client.query('SELECT version()');
console.log(res.rows[0]);
await client.end();
```
Troubleshooting tip:
✏️ Note that the container uses ephemeral storage, so all the data will be lost when the container is stopped.
This is intentional, as it simplifies the setup (and more importantly - the cleanup), while still allowing you to experiment and test your application.
* You can get the containter logs while tests are running by running `export DEBUG=testcontainers*` before running the tests.
### Python
Make sure you install `testcontainers` and `psycopg2` python packages:
```bash theme={null}
pip install testcontainers psycopg2-binary
```
and then you can run the following snippet:
```python theme={null}
from testcontainers.postgres import PostgresContainer
from testcontainers.core.waiting_utils import wait_for_logs
import psycopg
with PostgresContainer(
image="ghcr.io/niledatabase/testingcontainer:latest",
port=5432,
username="00000000-0000-0000-0000-000000000000",
password="password",
dbname="test",
driver="psycopg"
) as postgres:
delay = wait_for_logs(postgres, "Database has been created and is ready")
connection_url = f'postgresql://00000000-0000-0000-0000-000000000000:password@localhost:{postgres.get_exposed_port(5432)}/test'
client = psycopg.connect(connection_url)
with client.cursor() as cursor:
cursor.execute("SELECT version()")
print(cursor.fetchone())
```
Troubleshooting tip:
✏️ Note that the container uses ephemeral storage, so all the data will be lost when the container is stopped.
This is intentional, as it simplifies the setup (and more importantly - the cleanup), while still allowing you to experiment and test your application.
* On MacOS, you may need to explicitly set the `DOCKER_HOST` environment variable:
```bash theme={null}
export DOCKER_HOST=unix:///Users//.docker/run/docker.sock
```
Happy testing!
# Alembic
Source: https://thenile.dev/docs/getting-started/schema_migrations/alembic
Alembic is a migration tool for SQLAlchemy. It is a flexible tool for managing database migrations.
Below, we'll walk through the process of setting up Alembic in a project and running your first migration.
We'll start with an example project that has Alembic already set up, and you can
follow along with the steps below. Alternatively, you can run these steps in your own project.
## Running an Example Migration
```bash theme={null}
git clone https://github.com/niledatabase/niledatabase
cd niledatabase/examples/migrations/alembic
```
```bash theme={null}
virtualenv .venv
source .venv/bin/activate
# install alembic, sqlalchemy, psycopg2-binary and dotenv
pip install -r requirements.txt
# alembic init alembic # run this if you install alembic in your own project
```
In the example project, we have a migration script already created.
You can find it in the `./alembic/versions` directory.
Here's how to create a new migration script:
```bash theme={null}
alembic revision -m "create account table"
```
This will generate a new file: `./alembic/versions/13379be60997_create_account_table.py`. Now we want to edit the file and add the migration script to it:
```py theme={null}
def upgrade():
op.create_table(
'account',
sa.Column('id', sa.Integer, primary_key=True),
sa.Column('name', sa.String(50), nullable=False),
sa.Column('description', sa.Unicode(200)),
)
def downgrade():
op.drop_table('account')
```
To connect Alembic to the right database, create a `.env` file and add the connection string there:
```
DATABASE_URL=postgresql://user:password@host:5432/dbname
```
You can get your connection string from the Nile database home page.
```bash theme={null}
alembic upgrade head
```
Thats it! You can connect to your database and see the new table.
If you see an error that the table already exists, double check that you didn't accidentally create a second
migration with the `accounts` table on top of the existing one. If this happened to you, you can delete the new migration file (or alternatively, modify the table name).
## Generating Migrations
Alembic can also generate migrations for you. This is useful if you want to create a migration for a new table.
The example project has a `models.py` file that defines two models: `Todo` and `Tenant`.
To generate a migration for these models, you can use the following command:
```bash theme={null}
alembic revision --autogenerate -m "create todo and tenant models"
```
Alembic autogeneration compares your database state with models.py and will
drop any tables not defined there. Since Nile has a built-in `Tenants` table
that can't be dropped, we include the `Tenant` model in models.py to prevent
this.
You can then edit the generated migration file (if needed) and run it like as we did before:
```bash theme={null}
alembic upgrade head
```
## Next Steps
This is the most basic use-case of Alembic. There is a lot more to it, which you can learn
from the official [Alembic documentation](https://alembic.sqlalchemy.org/en/latest/front.html).
# Schema Migrations with Django
Source: https://thenile.dev/docs/getting-started/schema_migrations/django
Django has built-in support for schema migrations. Below, we'll walk through the process of setting up a Django project
with Nile and running your first migration.
If you don't already have Django installed, you can install it using pip:
```bash theme={null}
pip install django
```
```bash theme={null}
django-admin startproject myproject
```
This will create a new Django project in the `myproject` directory.
```bash theme={null}
cd myproject
python manage.py startapp myapp
```
This will create a new Django app in the `myproject` directory.
If all went well, the `myproject` directory should now have the following structure:
```
myproject/
manage.py
myproject/
__init__.py
settings.py
urls.py
wsgi.py
myapp/
__init__.py
admin.py
apps.py
migrations/
__init__.py
models.py
tests.py
views.py
```
Edit the `myproject/settings.py` file to include the Nile database URL:
```python theme={null}
DATABASES = {
"default": {
"ENGINE": "django.db.backends.postgresql",
"NAME": "mydb",
"USER": "myuser",
"PASSWORD": "mypassword",
"HOST": "us-west-2.db.thenile.dev",
"PORT": "5432",
}
}
```
You can find your database credentials in the Nile dashboard.
Django has several default modules that need database tables created. To create these tables, run the following command:
```bash theme={null}
python manage.py migrate
```
This will create the tables in the database.
In `myapp/models.py`, create a new model:
```python theme={null}
from django.db import models
class MyModel(models.Model):
name = models.CharField(max_length=255)
```
And update the installed apps in `myproject/settings.py` to include your app:
```python theme={null}
INSTALLED_APPS = [
'myapp.apps.MyappConfig',
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
]
```
To create a new migration, run the following command:
```bash theme={null}
python manage.py makemigrations myapp
```
This will create a new migration file in the `myapp/migrations` directory.
To run the migration, run the following command:
```bash theme={null}
python manage.py migrate
```
This will apply the migration to the database.
Connect to your database and see a new table created, called `myapp_mymodel`.
# Schema Migrations with Drizzle
Source: https://thenile.dev/docs/getting-started/schema_migrations/drizzle
Drizzle is a TypeScript ORM that supports Postgres, MySQL, and SQLite. It also has a CLI, `drizzle-kit`, for managing migrations and few other things.
This guide will show you how to use Drizzle Kit CLI to manage your schema migrations. We are going to assume that you already have a project set up with your
favorite Typescript framework.
Clone our example project and install dependencies:
```bash theme={null}
git clone https://github.com/niledatabase/niledatabase.git
cd examples/migrations/drizzle
npm i
```
This will install `drizzle-kit`, `drizzle-orm`, `dotenv`, and `pg-node` - all of which are needed for this guide. `pg-node` can be replaced with another postgres client like `postgres.js`.
To run this example, you'll need a .env file with a DATABASE\_URL environment variable set to a postgres database.
You can copy the connection string from your Nile database home page.
Drizzle kit is configured via a `drizzle.config.ts` file, which you can find in the root of the example project.
Here's an example `drizzle.config.ts` file. You'll need to set:
* The `schema` field to the path to your schema file
* The `out` field to the path where you want to store your migrations
* The `dialect` field to `postgresql` for Nile databases
* The `dbCredentials` field with your database credentials
```javascript theme={null}
import { defineConfig } from 'drizzle-kit';
import dotenv from 'dotenv';
dotenv.config();
export default defineConfig({
schema: './src/db/schema.ts',
out: './db/out',
dialect: 'postgresql',
dbCredentials: {
url:
process.env.DATABASE_URL ||
'postgresql://username:password@db.thenile.dev:5432/db',
},
});
```
In code-first schema management, you define your schema as Typescript objects, and then use the Drizzle Kit CLI to generate migrations.
Create your schema in `src/db/schema.ts`. Note that we include the built-in `tenants` table that Nile automatically provisions:
```javascript theme={null}
import { sql } from 'drizzle-orm';
import {
pgTable,
primaryKey,
uuid,
text,
timestamp,
varchar,
boolean,
vector,
} from 'drizzle-orm/pg-core';
export const tenants = pgTable('tenants', {
id: uuid('id')
.default(sql`gen_random_uuid()`)
.primaryKey(),
name: text('name'),
created: timestamp('created'),
updated: timestamp('updated'),
deleted: timestamp('deleted'),
});
export const todos = pgTable(
'todos',
{
id: uuid('id').default(sql`gen_random_uuid()`),
tenantId: uuid('tenant_id'),
title: varchar('title', { length: 256 }),
estimate: varchar('estimate', { length: 256 }),
embedding: vector('embedding', { dimensions: 768 }),
complete: boolean('complete'),
},
(table) => {
return {
pk: primaryKey({ columns: [table.tenantId, table.id] }),
};
},
);
```
Generate your first migration using Drizzle Kit:
```bash theme={null}
npx drizzle-kit generate
```
You should see output like:
```bash theme={null}
2 tables
tenants 5 columns 0 indexes 0 fks
todos 6 columns 0 indexes 0 fks
[✓] Your SQL migration file ➜ db/out/0000_absurd_captain_britain.sql 🚀
```
Then run the migration:
```bash theme={null}
npx drizzle-kit migrate
```
Now you can write code to insert and query data. Here's an example (`src/index.ts`):
```javascript theme={null}
import 'dotenv/config';
import { drizzle } from 'drizzle-orm/node-postgres';
import { todos, tenants } from './db/schema';
const db = drizzle(process.env.DATABASE_URL!);
async function main() {
const tenant: typeof tenants.$inferInsert = {
name: 'My New Tenant',
};
const tenantId = await db.insert(tenants).values(tenant).returning({ id: tenants.id });
console.log('New tenant created!')
const todo: typeof todos.$inferInsert = {
title: 'My New Todo',
tenantId: tenantId[0].id,
};
await db.insert(todos).values(todo);
console.log('New todo created!')
const rows = await db.select().from(todos);
console.log('Getting all todos from the database: ', rows)
}
main();
```
Run the example:
```bash theme={null}
npx tsx src/index.ts
```
You should see output showing the created tenant and todo:
```bash theme={null}
New tenant created!
New todo created!
Getting all todos from the database: [
{
id: 'd8896674-a7eb-4405-a4de-4ad6fbd2f5fc',
tenantId: '01929704-3250-70bf-9568-0a6858dfd4e9',
title: 'My New Todo',
estimate: null,
embedding: null,
complete: null
}
]
```
To add new columns, update your schema file. For example, to add a due date:
```javascript theme={null}
// ...
complete: boolean("complete"),
dueDate: timestamp("due_date"), // new column!
// ...
```
Generate and run a new migration:
```bash theme={null}
npx drizzle-kit generate
npx drizzle-kit migrate
```
This will generate a migration file like:
```sql theme={null}
ALTER TABLE "todos" ADD COLUMN "due_date" timestamp;
```
# Prisma
Source: https://thenile.dev/docs/getting-started/schema_migrations/prisma
Prisma is a popular ORM. It is a flexible tool for managing database migrations.
In this guide, we'll walk through the process of setting up Prisma in a project and
running several migration scenarios.
## Setting Up an New Prisma project
```bash theme={null}
mkdir hello-prisma
cd hello-prisma
# Initialize a new npm project
npm init -y
# Install and initialize typescript
npm install typescript tsx @types/node --save-dev
npx tsc --init
# Install and initialize Prisma
npm install prisma --save-dev
npx prisma init --datasource-provider postgresql
```
If you'd like to clone our example project, you can do so with the following command:
```bash theme={null}
git clone https://github.com/niledatabase/niledatabase
cd niledatabase/examples/migrations/prisma
```
```bash theme={null}
# Create a .env file
touch .env
# Add your database connection string to the .env file
echo "DATABASE_URL='postgresql://user:password@host:5432/dbname'" >> .env
```
You can get your connection string from the Nile console under the "Connections" page.
## Using Prisma with Nile in development
Nile has built-in tables that can be useful in your project.
Especially the `tenants` table, which is used to store tenant information.
Therefore, we recommend first pulling in the built-in tables to your project, and
then starting to build your own tables.
```bash theme={null}
npx prisma db pull
```
This will generate the `schema.prisma` file with the built-in `tenants` table.
```bash theme={null}
npx prisma generate
```
To add your own tables, you can edit the `schema.prisma` file and add your own models. For example:
```prisma theme={null}
model posts {
id String @default(dbgenerated("public.uuid_generate_v7()")) @db.Uuid
tenant_id String @db.Uuid
title String
content String?
authorId String
@@id([id, tenant_id])
}
```
```bash theme={null}
npx prisma db push
```
## Using Prisma with Nile in production
`db pull` and `db push` work well in development, where you can evolve the schema without tracking every change.
However, in production, we recommend using tracked migrations to ensure that the schema is always
in sync with the code.
Because Nile has a built-in `tenants` table, we first create a baseline migration that will include the built-in tables,
and then create additional migrations for our own tables.
If you don't yet have a schema.prisma file with the built-in tables, start by creating one:
```bash theme={null}
npx prisma db pull
```
Then, create a baseline migration:
```bash theme={null}
mkdir -p prisma/migrations/0_init
npx prisma migrate diff \
--from-empty \
--to-schema-datamodel prisma/schema.prisma \
--script > prisma/migrations/0_init/migration.sql
```
```bash theme={null}
npx prisma migrate resolve --applied 0_init
```
Now, lets modify the schema to include our own tables.
For example, we'll add a `posts` table to `schema.prisma`.
If you already have this table in your schema, you can add a column instead.
```prisma theme={null}
model posts {
id String @default(dbgenerated("public.uuid_generate_v7()")) @db.Uuid
tenant_id String @db.Uuid
title String
content String?
authorId String
@@id([id, tenant_id])
}
```
Now, we'll create a migration for this change by comparing the existing schema in the
DB (the `datasource`) with the new schema in `schema.prisma` (the `datamodel`).
While both the `datasource` and `datamodel` are in the same file, the first will refer to the
`datasource` definition in the `schema.prisma` file while the second will refer to the model itself.
```bash theme={null}
mkdir -p prisma/migrations/1_add_posts
npx prisma migrate diff \
--from-schema-datasource prisma/schema.prisma \
--to-schema-datamodel prisma/schema.prisma \
--script > prisma/migrations/1_add_posts/migration.sql
```
Prisma docs recommend generating migrations with `npx prisma migrate dev`. However, this will not work
for Nile because the `npx prisma migrate dev` command starts by attempting to "reset" a shadow database by dropping all tables.
This is not possible in Nile because the built-in tables are required for core functionality.
Last but not least, we can apply the migration to the database:
```bash theme={null}
npx prisma migrate deploy
```
# SmartBooks AI - Your Intelligent Accounting Partner
Source: https://thenile.dev/docs/getting-started/usecases/accounts
The accounting management application is designed to help organizations track their financial activities, including revenue and expenses. The application leverages AI to provide insights and suggestions for efficiency gains, trends analysis, and financial optimization. The system ensures compliance and security by associating all data with specific tenants, ensuring data isolation and integrity.
### Key Features:
1. **Revenue and Expense Tracking**:
* Track total revenue and expenses for each organization.
* Break down expenses by department.
2. **Document Upload and Parsing**:
* Upload invoices and receipts.
* Automatically parse documents to update revenue or expenses.
3. **AI-Powered Insights**:
* Analyze financial trends.
* Provide suggestions to fix increasing expenses or dropping revenue.
* Suggest efficiency gains by analyzing internal data and third-party vendor information.
4. **Reports and Analytics**:
* Generate detailed financial reports.
* View statistics and trends over time.
5. **User Management**:
* Support multiple roles including administrators, accountants, and auditors.
* Ensure secure access to financial data.
### Postgres Schemas
### 1. departments
Stores information about various departments within a tenant. This helps to report revenue and expenses by department.
```sql theme={null}
CREATE TABLE departments (
department_id UUID DEFAULT gen_random_uuid(),
tenant_id UUID,
department_name VARCHAR(100) NOT NULL,
PRIMARY KEY (tenant_id, department_id),
FOREIGN KEY (tenant_id) REFERENCES tenants(id)
);
```
### 2. revenue
Tracks revenue entries with amounts, dates, and descriptions. AI can automatically populate these fields with uploaded invoices, contracts and receipts.
```sql theme={null}
CREATE TABLE revenue (
tenant_id UUID,
revenue_id UUID DEFAULT gen_random_uuid(),
department_id UUID,
amount NUMERIC(15, 2) NOT NULL,
date TIMESTAMP NOT NULL,
description TEXT,
PRIMARY KEY (tenant_id, revenue_id),
FOREIGN KEY (tenant_id, department_id) REFERENCES departments(tenant_id, department_id)
);
```
### 3. expenses
Tracks expense entries with amounts, dates, and descriptions. AI can automatically populate these fields with uploaded invoices, contracts and receipts.
```sql theme={null}
CREATE TABLE expenses (
tenant_id UUID,
expense_id UUID DEFAULT gen_random_uuid(),
department_id UUID,
amount NUMERIC(15, 2) NOT NULL,
date TIMESTAMP NOT NULL,
description TEXT,
PRIMARY KEY (tenant_id, expense_id),
FOREIGN KEY (tenant_id, department_id) REFERENCES departments(tenant_id, department_id)
);
```
### 4. invoices
Stores uploaded invoices, including file paths and associated amounts.The invoice itself is stored where the file\_path is referencing but the embeddings are generated for them and stored in the table. In a real world application, these invoices will be chunked and embeddings will be generated for each chunk.
```sql theme={null}
CREATE TABLE invoices (
tenant_id UUID,
invoice_id UUID DEFAULT gen_random_uuid(),
department_id UUID,
file_path TEXT NOT NULL,
amount NUMERIC(15, 2) NOT NULL,
date TIMESTAMP NOT NULL,
description TEXT,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
PRIMARY KEY (tenant_id, invoice_id),
FORE
```
### 5. receipts
Stores uploaded receipts, including file paths and associated amounts. The receipt itself is stored where the file\_path is referencing but the embeddings are generated for them and stored in the table. In a real world application, these receipts will be chunked and embeddings will be generated for each chunk.
```sql theme={null}
CREATE TABLE receipts (
tenant_id UUID,
receipt_id UUID DEFAULT gen_random_uuid(),
department_id UUID,
file_path TEXT NOT NULL,
amount NUMERIC(15, 2) NOT NULL,
date TIMESTAMP NOT NULL,
description TEXT,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
PRIMARY KEY (tenant_id, receipt_id),
FOREIGN KEY (tenant_id, department_id) REFERENCES departments(tenant_id, department_id)
);
```
### 6. financial\_reports
Stores generated financial reports, including type and data in JSON format.These financial report are also pushed to the AI model to help with answering questions about the reports.
```sql theme={null}
CREATE TABLE financial_reports (
tenant_id UUID,
report_id UUID DEFAULT gen_random_uuid(),
report_type VARCHAR(50), -- e.g., "monthly", "quarterly", "annual"
generated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
data JSONB,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
PRIMARY KEY (tenant_id, report_id),
FOREIGN KEY (tenant_id) REFERENCES tenants(id)
);
```
### Full Script
```sql theme={null}
-- Create Departments Table
CREATE TABLE departments (
department_id UUID DEFAULT gen_random_uuid(),
tenant_id UUID,
department_name VARCHAR(100) NOT NULL,
PRIMARY KEY (tenant_id, department_id),
FOREIGN KEY (tenant_id) REFERENCES tenants(id)
);
-- Create Revenue Table
CREATE TABLE revenue (
tenant_id UUID,
revenue_id UUID DEFAULT gen_random_uuid(),
department_id UUID,
amount NUMERIC(15, 2) NOT NULL,
date TIMESTAMP NOT NULL,
description TEXT,
PRIMARY KEY (tenant_id, revenue_id),
FOREIGN KEY (tenant_id, department_id) REFERENCES departments(tenant_id, department_id)
);
-- Create Expenses Table
CREATE TABLE expenses (
tenant_id UUID,
expense_id UUID DEFAULT gen_random_uuid(),
department_id UUID,
amount NUMERIC(15, 2) NOT NULL,
date TIMESTAMP NOT NULL,
description TEXT,
PRIMARY KEY (tenant_id, expense_id),
FOREIGN KEY (tenant_id, department_id) REFERENCES departments(tenant_id, department_id)
);
-- Create Invoices Table
CREATE TABLE invoices (
tenant_id UUID,
invoice_id UUID DEFAULT gen_random_uuid(),
department_id UUID,
file_path TEXT NOT NULL,
amount NUMERIC(15, 2) NOT NULL,
date TIMESTAMP NOT NULL,
description TEXT,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
PRIMARY KEY (tenant_id, invoice_id),
FOREIGN KEY (tenant_id, department_id) REFERENCES departments(tenant_id, department_id)
);
-- Create Receipts Table
CREATE TABLE receipts (
tenant_id UUID,
receipt_id UUID DEFAULT gen_random_uuid(),
department_id UUID,
file_path TEXT NOT NULL,
amount NUMERIC(15, 2) NOT NULL,
date TIMESTAMP NOT NULL,
description TEXT,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
PRIMARY KEY (tenant_id, receipt_id),
FOREIGN KEY (tenant_id, department_id) REFERENCES departments(tenant_id, department_id)
);
-- Create Financial Reports Table
CREATE TABLE financial_reports (
tenant_id UUID,
report_id UUID DEFAULT gen_random_uuid(),
report_type VARCHAR(50), -- e.g., "monthly", "quarterly", "annual"
generated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
data JSONB,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
PRIMARY KEY (tenant_id, report_id),
FOREIGN KEY (tenant_id) REFERENCES tenants(id)
);
```
# SmartSpend AI - Set teams free from manual expenses
Source: https://thenile.dev/docs/getting-started/usecases/expensify
The AI-native expense management system will be designed to automate the tracking, submission, and analysis of expenses using an intelligent AI agent. This agent will be capable of understanding and processing natural language inputs, allowing users to submit expenses through a chat interface. The system will support generative AI techniques using Retrieval-Augmented Generation (RAG) to provide insights and search through past expenses effectively. It will also leverage vector embeddings to store and query expense-related data, ensuring efficient and accurate information retrieval. The system will track expenses per tenant, making tenant\_id a key field in every schema. The AI agent will process receipts, categorize expenses, flag anomalies, and generate expense reports. It also provides analytics and insights on spending patterns, helping users make informed financial decisions.
### Postgres Schemas
### 1. expenses
Tracks individual expenses submitted by users.
```sql theme={null}
CREATE TABLE expenses (
expense_id uuid,
tenant_id uuid,
user_id uuid,
amount DECIMAL(10, 2),
category VARCHAR(255),
description TEXT,
expense_date DATE,
expense_embedding vector(256),
PRIMARY KEY(tenant_id, expense_id),
FOREIGN KEY (tenant_id) REFERENCES tenants(id)
);
```
### 2. receipts
Stores receipt images and links them to expenses.
```sql theme={null}
CREATE TABLE receipts (
receipt_id uuid,
tenant_id uuid,
expense_id uuid,
receipt_image BYTEA,
receipts_embedding vector(256),
PRIMARY KEY (tenant_id, receipt_id),
FOREIGN KEY (tenant_id, expense_id) REFERENCES expenses(tenant_id, expense_id)
);
```
### 3. expense\_reports
Aggregates expenses into reports for review and approval.
```sql theme={null}
CREATE TABLE expense_reports (
report_id uuid,
tenant_id uuid,
user_id uuid,
report_name VARCHAR(255),
start_date DATE,
end_date DATE,
status VARCHAR(50),
PRIMARY KEY (tenant_id, report_id),
FOREIGN KEY (tenant_id, user_id) REFERENCES users.tenant_users(tenant_id, user_id)
);
```
### 4. expense\_report\_details
Links expenses to expense reports.
```sql theme={null}
CREATE TABLE expense_report_details (
report_detail_id uuid,
report_id uuid,
expense_id uuid,
tenant_id uuid,
PRIMARY KEY (tenant_id, report_detail_id),
FOREIGN KEY (tenant_id, expense_id) REFERENCES expenses(tenant_id, expense_id),
FOREIGN KEY (tenant_id, report_id) REFERENCES expense_reports(tenant_id, report_id)
);
```
### 5. user\_preferences
Stores user preferences for personalized AI responses.
```sql theme={null}
CREATE TABLE user_preferences (
preference_id uuid,
tenant_id uuid,
user_id uuid,
preference_data JSONB,
PRIMARY KEY (tenant_id, preference_id),
FOREIGN KEY (tenant_id, user_id) REFERENCES users.tenant_users(tenant_id, user_id)
);
```
### 6. expense\_analytics
Stores analytical data and insights generated by the AI.
```sql theme={null}
CREATE TABLE expense_analytics (
analytics_id uuid,
tenant_id uuid,
user_id uuid,
analysis_date DATE,
insights TEXT,
PRIMARY KEY (tenant_id, analytics_id),
FOREIGN KEY (tenant_id, user_id) REFERENCES users.tenant_users(tenant_id, user_id)
);
```
### Full Script
```sql theme={null}
CREATE TABLE expenses (
expense_id uuid,
tenant_id uuid,
user_id uuid,
amount DECIMAL(10, 2),
category VARCHAR(255),
description TEXT,
expense_date DATE,
expense_embedding vector(256),
PRIMARY KEY(tenant_id, expense_id),
FOREIGN KEY (tenant_id) REFERENCES tenants(id)
);
CREATE TABLE receipts (
receipt_id uuid,
tenant_id uuid,
expense_id uuid,
receipt_image BYTEA,
receipts_embedding vector(256),
PRIMARY KEY (tenant_id, receipt_id),
FOREIGN KEY (tenant_id, expense_id) REFERENCES expenses(tenant_id, expense_id)
);
CREATE TABLE expense_reports (
report_id uuid,
tenant_id uuid,
user_id uuid,
report_name VARCHAR(255),
start_date DATE,
end_date DATE,
status VARCHAR(50),
PRIMARY KEY (tenant_id, report_id),
FOREIGN KEY (tenant_id, user_id) REFERENCES users.tenant_users(tenant_id, user_id)
);
CREATE TABLE expense_report_details (
report_detail_id uuid,
report_id uuid,
expense_id uuid,
tenant_id uuid,
PRIMARY KEY (tenant_id, report_detail_id),
FOREIGN KEY (tenant_id, expense_id) REFERENCES expenses(tenant_id, expense_id),
FOREIGN KEY (tenant_id, report_id) REFERENCES expense_reports(tenant_id, report_id)
);
CREATE TABLE user_preferences (
preference_id uuid,
tenant_id uuid,
user_id uuid,
preference_data JSONB,
PRIMARY KEY (tenant_id, preference_id),
FOREIGN KEY (tenant_id, user_id) REFERENCES users.tenant_users(tenant_id, user_id)
);
CREATE TABLE expense_analytics (
analytics_id uuid,
tenant_id uuid,
user_id uuid,
analysis_date DATE,
insights TEXT,
PRIMARY KEY (tenant_id, analytics_id),
FOREIGN KEY (tenant_id, user_id) REFERENCES users.tenant_users(tenant_id, user_id)
);
```
# Feedback360 AI - 360-Degree Feedback, Perfected with AI
Source: https://thenile.dev/docs/getting-started/usecases/feedback
The 360 employee feedback application allows employees to provide feedback about their peers, receive feedback, and manage their own performance and recognition. The application supports feedback from any employee to any other employee within the company and includes AI-powered features to analyze feedback patterns, draft feedback, and provide insights. Employees can also give kudos and badges to each other, with options for anonymous feedback. The HR team can manage and review all feedback, with notifications to ensure timely completion of feedback tasks.
### Key Features
1. **Feedback Management:**
* Employees can give feedback to any other employee.
* Feedback can be given anonymously if desired.
* Employees can view all feedback given to them.
* Self-feedback is allowed.
2. **AI Features:**
* AI can summarize feedback and provide insights into feedback patterns for specific employees.
* AI can help draft feedback based on context or previous feedback.
* AI can analyze feedback trends to suggest improvements or recognize strengths.
3. **Recognition:**
* Employees can give kudos and badges to each other.
* System tracks and manages all kudos and badges given.
4. **Notifications:**
* Notifications for employees to provide pending feedback.
* Alerts for HR and employees regarding feedback deadlines.
5. **HR Management:**
* HR can view and manage all feedback across the organization.
* HR can use AI to summarize and analyze overall feedback trends.
### Postgres Schemas
### 1. departments
The `departments` table contains information about the different departments within an organization. Each department is uniquely identified by a `department_id` and is associated with a `tenant_id` to ensure it belongs to the correct organization. This table includes the department name.
```sql theme={null}
CREATE TABLE departments (
tenant_id UUID,
department_id UUID,
name VARCHAR(100) NOT NULL,
PRIMARY KEY (tenant_id, department_id),
FOREIGN KEY (tenant_id) REFERENCES tenants(tenant_id)
);
```
### 2. employees
The `employees` table tracks employee information within an organization. Each employee is identified by a unique `employee_id` and is associated with a department (`department_id`). This table includes details such as job title and compensation. The table is linked to the `tenant_id` to ensure that employee records are specific to each organization.
```sql theme={null}
CREATE TABLE employees (
tenant_id UUID,
employee_id UUID,
department_id UUID,
title VARCHAR(100),
compensation NUMERIC(15, 2),
PRIMARY KEY (tenant_id, employee_id),
FOREIGN KEY (tenant_id, employee_id) REFERENCES tenant_users(tenant_id, user_id),
FOREIGN KEY (tenant_id, department_id) REFERENCES departments(tenant_id, department_id)
);
```
### 3. feedback
The `feedback` table records feedback given by employees to other employees. It includes the `giver_id` (the employee giving the feedback) and `receiver_id` (the employee receiving the feedback), along with the feedback text. This table also tracks whether the feedback is anonymous and includes a `vector_embedding` for AI-based analysis. Each feedback entry is tied to a specific `tenant_id`. The embeddings can be used to help summarize and draft feedbacks.
```sql theme={null}
CREATE TABLE feedback (
tenant_id UUID,
feedback_id UUID,
giver_id UUID,
receiver_id UUID,
feedback_text TEXT,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
is_anonymous BOOLEAN DEFAULT FALSE,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
PRIMARY KEY (tenant_id, feedback_id),
FOREIGN KEY (tenant_id, giver_id) REFERENCES employees(tenant_id, employee_id),
FOREIGN KEY (tenant_id, receiver_id) REFERENCES employees(tenant_id, employee_id)
);
```
### 4. self\_feedback
The `self_feedback` table allows employees to provide feedback about themselves. Each record includes a unique `self_feedback_id`, the `employee_id` who provided the feedback, and the text of the feedback. This table includes a `vector_embedding` for AI analysis and is associated with a specific `tenant_id`. The embeddings can be used to help summarize and draft feedbacks.
```sql theme={null}
CREATE TABLE self_feedback (
tenant_id UUID,
self_feedback_id UUID,
employee_id UUID,
feedback_text TEXT,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, self_feedback_id),
FOREIGN KEY (tenant_id, employee_id) REFERENCES employees(tenant_id, employee_id)
);
```
### 5. kudos
The `kudos` table tracks kudos given by employees to their peers. It includes a unique `kudos_id`, the `giver_id` (employee giving kudos), the `receiver_id` (employee receiving kudos), and the text of the kudos. A `vector_embedding` is included for AI analysis. The embeddings are used to help search past kudos by HR and understand correlation between feedback and kudos. Each entry is linked to a `tenant_id`.
```sql theme={null}
CREATE TABLE kudos (
tenant_id UUID,
kudos_id UUID,
giver_id UUID,
receiver_id UUID,
kudos_text TEXT,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
PRIMARY KEY (tenant_id, kudos_id),
FOREIGN KEY (tenant_id, giver_id) REFERENCES employees(tenant_id, employee_id),
FOREIGN KEY (tenant_id, receiver_id) REFERENCES employees(tenant_id, employee_id)
);
```
### 6. badges
The `badges` table records badges awarded to employees. Each badge has a unique `badge_id`, is linked to an `employee_id`, and includes the badge's name and the date it was issued. This table helps track employee achievements and is specific to each `tenant_id`.
```sql theme={null}
CREATE TABLE badges (
tenant_id UUID,
badge_id UUID,
employee_id UUID,
badge_name VARCHAR(100),
issued_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, badge_id),
FOREIGN KEY (tenant_id, employee_id) REFERENCES employees(tenant_id, employee_id)
);
```
### 7. notifications
The `notifications` table manages notifications sent to employees. Each notification includes a unique `notification_id`, the `employee_id` it is intended for, the text of the notification, and a timestamp for when it was created. The table also tracks whether the notification has been read. This table is linked to a specific `tenant_id`.
```sql theme={null}
CREATE TABLE notifications (
tenant_id UUID,
notification_id UUID,
employee_id UUID,
notification_text TEXT,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
is_read BOOLEAN DEFAULT FALSE,
PRIMARY KEY (tenant_id, notification_id),
FOREIGN KEY (tenant_id, employee_id) REFERENCES employees(tenant_id, employee_id)
);
```
### Full Script
```sql theme={null}
-- Departments Table
CREATE TABLE departments (
tenant_id UUID,
department_id UUID,
name VARCHAR(100) NOT NULL,
PRIMARY KEY (tenant_id, department_id),
FOREIGN KEY (tenant_id) REFERENCES tenants(id)
);
-- Employees Table
CREATE TABLE employees (
tenant_id UUID,
employee_id UUID,
department_id UUID,
title VARCHAR(100),
compensation NUMERIC(15, 2),
PRIMARY KEY (tenant_id, employee_id),
FOREIGN KEY (tenant_id, employee_id) REFERENCES users.tenant_users(tenant_id, user_id),
FOREIGN KEY (tenant_id, department_id) REFERENCES departments(tenant_id, department_id)
);
-- Feedback Table
CREATE TABLE feedback (
tenant_id UUID,
feedback_id UUID,
giver_id UUID,
receiver_id UUID,
feedback_text TEXT,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
is_anonymous BOOLEAN DEFAULT FALSE,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
PRIMARY KEY (tenant_id, feedback_id),
FOREIGN KEY (tenant_id, giver_id) REFERENCES employees(tenant_id, employee_id),
FOREIGN KEY (tenant_id, receiver_id) REFERENCES employees(tenant_id, employee_id)
);
-- Self Feedback Table
CREATE TABLE self_feedback (
tenant_id UUID,
self_feedback_id UUID,
employee_id UUID,
feedback_text TEXT,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, self_feedback_id),
FOREIGN KEY (tenant_id, employee_id) REFERENCES employees(tenant_id, employee_id)
);
-- Kudos Table
CREATE TABLE kudos (
tenant_id UUID,
kudos_id UUID,
giver_id UUID,
receiver_id UUID,
kudos_text TEXT,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
PRIMARY KEY (tenant_id, kudos_id),
FOREIGN KEY (tenant_id, giver_id) REFERENCES employees(tenant_id, employee_id),
FOREIGN KEY (tenant_id, receiver_id) REFERENCES employees(tenant_id, employee_id)
);
-- Badges Table
CREATE TABLE badges (
tenant_id UUID,
badge_id UUID,
employee_id UUID,
badge_name VARCHAR(100),
issued_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, badge_id),
FOREIGN KEY (tenant_id, employee_id) REFERENCES employees(tenant_id, employee_id)
);
-- Notifications Table
CREATE TABLE notifications (
tenant_id UUID,
notification_id UUID,
employee_id UUID,
notification_text TEXT,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
is_read BOOLEAN DEFAULT FALSE,
PRIMARY KEY (tenant_id, notification_id),
FOREIGN KEY (tenant_id, employee_id) REFERENCES employees(tenant_id, employee_id)
);
```
# HealthPilot - Enhancing Patient Care with Intelligent Assistance
Source: https://thenile.dev/docs/getting-started/usecases/health
The Healthcare Patient Management Application is designed to facilitate the efficient management of patient information within a healthcare setting. It provides functionality for adding new patients, tracking their personal information, medical history, current appointment notes, scheduling future appointments, and managing test results. Additionally, the application leverages AI to assist doctors by summarizing patient medical states, diagnosing potential conditions, and analyzing test result trends.
### Key Features
1. **Patient Management:**
* **Add New Patients:** Capture and store detailed patient personal information.
* **View Patient Details:** Access comprehensive patient profiles, including personal details and medical history.
* **Edit Patient Information:** Update patient records as needed.
2. **Medical Records:**
* **Track Medical History:** Store and retrieve past medical records.
* **Appointment Notes:** Document notes from current and past appointments.
* **Schedule Appointments:** Track upcoming appointments and notify patients.
* **Test Results:** Store and retrieve test results associated with medical records.
3. **Department Management:**
* **Track Departments:** Manage information about various departments within the healthcare organization.
* **Assign Doctors to Departments:** Ensure each doctor is associated with a specific department.
4. **AI Assistance:**
* **Summarize Medical State:** Summarize patient medical information based on past records.
* **Diagnosis Assistance:** Provide possible diagnoses based on symptoms and medical history.
* **Search and Analyze:** Search through test results and analyze trends in patient data.
5. **Security and Compliance:**
* **Data Privacy:** Ensure patient data is stored securely and complies with relevant healthcare regulations.
* **Access Control:** Restrict access to sensitive information based on user roles.
### Postgres Schemas
### 1. patients
Stores patient information, including personal details and medical history. The medical history consist of any known issues that the patient has provided. This information is useful when diagnosing issues. The vector embeddings are calculated on the medical history to use it to ask AI about the patients, diagnose problems given specific symptoms and even identify tests that needs to be done.
```sql theme={null}
CREATE TABLE patients (
tenant_id UUID,
patient_id UUID DEFAULT gen_random_uuid(),
first_name VARCHAR(100) NOT NULL,
last_name VARCHAR(100) NOT NULL,
date_of_birth DATE NOT NULL,
gender VARCHAR(10),
contact_info JSONB,
medical_history TEXT,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, patient_id),
FOREIGN KEY (tenant_id) REFERENCES tenants(id)
);
```
### 2. departments
Stores information about departments within the healthcare organization.
```sql theme={null}
CREATE TABLE departments (
tenant_id UUID,
department_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
department_name VARCHAR(100),
description TEXT,
FOREIGN KEY (tenant_id) REFERENCES tenants(tenant_id)
);
```
### 3. doctors
Stores information about doctors, including their specializations and contact details. A doctor will mostly belong to a s pecific department.
```sql theme={null}
CREATE TABLE doctors (
tenant_id UUID,
doctor_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
first_name VARCHAR(100) NOT NULL,
last_name VARCHAR(100) NOT NULL,
specialization VARCHAR(100),
contact_info JSONB,
department_id UUID,
FOREIGN KEY (tenant_id, doctor_id) REFERENCES tenant_users(tenant_id, user_id),
FOREIGN KEY (tenant_id, department_id) REFERENCES departments(tenant_id, department_id)
);
```
### 4. medical\_records
Stores detailed medical records for each patient. There is one medical record for each health problem the patient comes to visit the doctor. The medical records also contains a list of tests taken, results and any other notes. The embeddings are calculated on the medical record data to be able to help with proposed treatments, searching past records and even correlate issues across patients using AI (given permission).
```sql theme={null}
CREATE TABLE medical_records (
tenant_id UUID,
record_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
patient_id UUID,
record_date TIMESTAMP,
description TEXT,
record_data JSONB,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
FOREIGN KEY (tenant_id, patient_id) REFERENCES patients(tenant_id, patient_id)
);
```
### 5. test\_results
Stores test results associated with medical records. The test results are also stored as embeddings to use it with an AI model to identify correlation of results with symptoms, propose treatments and show trends.
```sql theme={null}
CREATE TABLE test_results (
tenant_id UUID,
test_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
record_id UUID,
test_name VARCHAR(100),
test_date TIMESTAMP,
results JSONB,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
FOREIGN KEY (tenant_id, record_id) REFERENCES medical_records(tenant_id, record_id)
);
```
### 6. appointments
Tracks appointments for patients, including notes and status.
```sql theme={null}
CREATE TABLE appointments (
tenant_id UUID,
appointment_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
patient_id UUID,
appointment_date TIMESTAMP,
doctor_id UUID,
notes TEXT,
status VARCHAR(50),
FOREIGN KEY (tenant_id, patient_id) REFERENCES patients(tenant_id, patient_id),
FOREIGN KEY (tenant_id, doctor_id) REFERENCES tenant_users(tenant_id, user_id)
);
```
### Full Script
```sql theme={null}
-- Patients Table
-- Description: Stores patient information, including personal details and medical history.
CREATE TABLE patients (
tenant_id UUID,
patient_id UUID DEFAULT gen_random_uuid(),
first_name VARCHAR(100) NOT NULL,
last_name VARCHAR(100) NOT NULL,
date_of_birth DATE NOT NULL,
gender VARCHAR(10),
contact_info JSONB,
medical_history TEXT,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, patient_id),
FOREIGN KEY (tenant_id) REFERENCES tenants(id)
);
-- Departments Table
-- Description: Stores information about departments within the healthcare organization.
CREATE TABLE departments (
tenant_id UUID,
department_id UUID DEFAULT gen_random_uuid(),
department_name VARCHAR(100),
description TEXT,
PRIMARY KEY (tenant_id, department_id),
FOREIGN KEY (tenant_id) REFERENCES tenants(id)
);
-- Doctors Table
-- Description: Stores information about doctors, including their specializations and contact details.
CREATE TABLE doctors (
tenant_id UUID,
doctor_id UUID DEFAULT gen_random_uuid(),
first_name VARCHAR(100) NOT NULL,
last_name VARCHAR(100) NOT NULL,
specialization VARCHAR(100),
contact_info JSONB,
department_id UUID,
PRIMARY KEY (tenant_id, doctor_id),
FOREIGN KEY (tenant_id, doctor_id) REFERENCES users.tenant_users(tenant_id, user_id),
FOREIGN KEY (tenant_id, department_id) REFERENCES departments(tenant_id, department_id)
);
-- Medical Records Table
-- Description: Stores detailed medical records for each patient.
CREATE TABLE medical_records (
tenant_id UUID,
record_id UUID DEFAULT gen_random_uuid(),
patient_id UUID,
record_date TIMESTAMP,
description TEXT,
record_data JSONB,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
PRIMARY KEY (tenant_id, record_id),
FOREIGN KEY (tenant_id, patient_id) REFERENCES patients(tenant_id, patient_id)
);
-- Test Results Table
-- Description: Stores test results associated with medical records.
CREATE TABLE test_results (
tenant_id UUID,
test_id UUID DEFAULT gen_random_uuid(),
record_id UUID,
test_name VARCHAR(100),
test_date TIMESTAMP,
results JSONB,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
PRIMARY KEY (tenant_id, test_id),
FOREIGN KEY (tenant_id, record_id) REFERENCES medical_records(tenant_id, record_id)
);
-- Appointments Table
-- Description: Tracks appointments for patients, including notes and status.
CREATE TABLE appointments (
tenant_id UUID,
appointment_id UUID DEFAULT gen_random_uuid(),
patient_id UUID,
appointment_date TIMESTAMP,
doctor_id UUID,
notes TEXT,
status VARCHAR(50),
PRIMARY KEY (tenant_id, appointment_id),
FOREIGN KEY (tenant_id, patient_id) REFERENCES patients(tenant_id, patient_id),
FOREIGN KEY (tenant_id, doctor_id) REFERENCES users.tenant_users(tenant_id, user_id)
);
```
# RecruitAI - Smarter Hiring, Powered by AI
Source: https://thenile.dev/docs/getting-started/usecases/hiring
The multitenant recruiting application will streamline the hiring process for recruiters and hiring managers, incorporating AI features to enhance candidate sourcing, interview scheduling, and feedback management. Below are the detailed requirements and the necessary PostgreSQL schemas.
### Key Features
1. **Recruiter Features:**
* **Add Candidates:** Recruiters can manually add candidate profiles to different open roles.
* **AI Candidate Sourcing:** Recruiters can input job descriptions, and the AI will automatically find and add candidate profiles from online sources.
* **Interview Scheduling:** Recruiters can schedule interviews for candidates or let the AI handle scheduling.
* **Status Tracking:** Recruiters can update the status of candidates throughout the hiring process.
2. **Hiring Manager Features:**
* **View Resumes:** Hiring managers can view candidate resumes and profiles.
* **Provide Feedback:** Hiring managers can add feedback for candidates, which is tracked and visible to the recruiting team.
* **AI Resume Query:** Hiring managers can use an AI chat interface to query candidate resumes and get summaries.
* **Generate Interview Questions:** Based on job descriptions, the AI can generate tailored interview questions.
3. **AI Features:**
* **Candidate Sourcing:** Automatically find candidates online based on job descriptions.
* **Interview Scheduling:** Optimize interview schedules based on availability.
* **Profile Summarization:** Summarize candidate profiles and resumes for quick insights.
* **Interview Question Generation:** Generate relevant interview questions based on job descriptions and candidate profiles.
### PostgreSQL Schemas
### 1. candidates
This table tracks all the candidates being considered for hiring, storing personal information, contact details, and their resumes. Each candidate is associated with a tenant.
```sql theme={null}
CREATE TABLE candidates (
tenant_id UUID,
candidate_id UUID DEFAULT gen_random_uuid(),
name VARCHAR(100) NOT NULL,
email VARCHAR(100) UNIQUE NOT NULL,
phone VARCHAR(20),
resume TEXT NOT NULL, -- assuming resume is stored as text
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, candidate_id),
FOREIGN KEY (tenant_id) REFERENCES tenants(id)
);
```
### 2. recruiter
This table contains information about recruiters, including their conversion success rates. It references the tenant and user details from the tenant\_users table.
```sql theme={null}
CREATE TABLE recruiter (
tenant_id UUID,
recruiter_id UUID,
conversion_success INT NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, recruiter_id),
FOREIGN KEY (tenant_id, recruiter_id) REFERENCES tenant_users(tenant_id, user_id)
);
```
### 3. hiring\_manager
This table stores information about hiring managers, linking them to specific tenants and user accounts. It helps identify the hiring managers responsible for job postings.
```sql theme={null}
CREATE TABLE hiring_manager (
tenant_id UUID,
manager_id UUID,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, manager_id),
FOREIGN KEY (tenant_id, manager_id) REFERENCES tenant_users(tenant_id, user_id)
);
```
### 4. jobs
This table tracks all open job positions, including the job title, description, and status. Each job is assigned to a hiring manager within a specific tenant's organization. The job description is used to calculate vector embeddings. This can be used to generate interview questions, search for matching candidates and even ask questions to a chatbot about jobs in the system.
```sql theme={null}
CREATE TABLE jobs (
tenant_id UUID,
job_id UUID DEFAULT gen_random_uuid(),
title VARCHAR(100) NOT NULL,
description TEXT,
status VARCHAR(50) DEFAULT 'open', -- e.g., open, closed, in_progress
hiring_manager_id UUID,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
PRIMARY KEY (tenant_id, job_id),
FOREIGN KEY (tenant_id, hiring_manager_id) REFERENCES hiring_manager(tenant_id, manager_id)
);
```
### 5. candidate\_jobs
This table maps candidates to the jobs they have applied for or been considered for, tracking the status of their application for each job.
```sql theme={null}
CREATE TABLE candidate_jobs (
tenant_id UUID,
candidate_id UUID,
job_id UUID,
status VARCHAR(50) DEFAULT 'applied', -- status of the candidate for this job
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, candidate_id, job_id),
FOREIGN KEY (tenant_id, candidate_id) REFERENCES candidates(tenant_id, candidate_id),
FOREIGN KEY (tenant_id, job_id) REFERENCES jobs(tenant_id, job_id)
);
```
### 6. interviews
This table manages the interview schedules for each job, including interview dates, questions, and expected answers. It helps organize the interview process for candidates. The questions and expected answers are converted to embeddings to help search for similar questions or similar answers. This helps to use the AI to create new questions of similar difficulty or correct answers from candidates.
```sql theme={null}
CREATE TABLE interviews (
tenant_id UUID,
interview_id UUID DEFAULT gen_random_uuid(),
job_id UUID,
interview_date TIMESTAMP NOT NULL,
questions JSONB NOT NULL, -- storing interview questions in JSONB format
expected_answers JSONB, -- storing expected answers in JSONB format
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, interview_id),
FOREIGN KEY (tenant_id, job_id) REFERENCES jobs(tenant_id, job_id)
);
```
### 7. feedback
This table captures feedback provided by users (recruiters or hiring managers) for each interview. It includes detailed feedback text and links to the specific interview and user who provided it. The feedback are converted to embeddings to be able to search for similar feedbacks in the system and check what scores have been given by other recruiters and managers. This helps to calibrate how a particular feedback and rating correlate.
```sql theme={null}
CREATE TABLE feedback (
tenant_id UUID,
feedback_id UUID DEFAULT gen_random_uuid(),
interview_id UUID,
user_id UUID,
feedback_text TEXT NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
PRIMARY KEY (tenant_id, feedback_id),
FOREIGN KEY (tenant_id, interview_id) REFERENCES interviews(tenant_id, interview_id),
FOREIGN KEY (tenant_id, user_id) REFERENCES tenant_users(tenant_id, user_id)
);
◦
```
### Complete SQL Script
```sql theme={null}
CREATE TABLE candidates (
tenant_id UUID,
candidate_id UUID DEFAULT gen_random_uuid(),
name VARCHAR(100) NOT NULL,
email VARCHAR(100) NOT NULL,
phone VARCHAR(20),
resume TEXT NOT NULL, -- assuming resume is stored as text
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, candidate_id),
FOREIGN KEY (tenant_id) REFERENCES tenants(id)
);
CREATE TABLE recruiter (
tenant_id UUID,
recruiter_id UUID,
conversion_success INT NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, recruiter_id),
FOREIGN KEY (tenant_id, recruiter_id) REFERENCES tenant_users(tenant_id, user_id)
);
CREATE TABLE hiring_manager (
tenant_id UUID,
manager_id UUID,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, manager_id),
FOREIGN KEY (tenant_id, manager_id) REFERENCES tenant_users(tenant_id, user_id)
);
CREATE TABLE jobs (
tenant_id UUID,
job_id UUID DEFAULT gen_random_uuid(),
title VARCHAR(100) NOT NULL,
description TEXT,
status VARCHAR(50) DEFAULT 'open', -- e.g., open, closed, in_progress
hiring_manager_id UUID,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
PRIMARY KEY (tenant_id, job_id),
FOREIGN KEY (tenant_id, hiring_manager_id) REFERENCES hiring_manager(tenant_id, manager_id)
);
CREATE TABLE candidate_jobs (
tenant_id UUID,
candidate_id UUID,
job_id UUID,
status VARCHAR(50) DEFAULT 'applied', -- status of the candidate for this job
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, candidate_id, job_id),
FOREIGN KEY (tenant_id, candidate_id) REFERENCES candidates(tenant_id, candidate_id),
FOREIGN KEY (tenant_id, job_id) REFERENCES jobs(tenant_id, job_id)
);
CREATE TABLE interviews (
tenant_id UUID,
interview_id UUID DEFAULT gen_random_uuid(),
job_id UUID,
interview_date TIMESTAMP NOT NULL,
questions JSONB NOT NULL, -- storing interview questions in JSONB format
expected_answers JSONB, -- storing expected answers in JSONB format
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, interview_id),
FOREIGN KEY (tenant_id, job_id) REFERENCES jobs(tenant_id, job_id)
);
CREATE TABLE feedback (
tenant_id UUID,
feedback_id UUID DEFAULT gen_random_uuid(),
interview_id UUID,
user_id UUID,
feedback_text TEXT NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
PRIMARY KEY (tenant_id, feedback_id),
FOREIGN KEY (tenant_id, interview_id) REFERENCES interviews(tenant_id, interview_id),
FOREIGN KEY (tenant_id, user_id) REFERENCES tenant_users(tenant_id, user_id)
);
```
# HRIntelli - Magically Simplifying HR with AI
Source: https://thenile.dev/docs/getting-started/usecases/hr
The HR management application will now include features to manage departments and employee benefits. Each department will have its own information and employees will be associated with a department. Additionally, employee benefits will be tracked.
### Key Feature:
1. **Employee Management**:
* Add new employees with their personal details, title, and compensation.
* Update employee details, including title changes, personal details, and compensation adjustments.
* Terminate employees and update their status.
2. **Department Management**:
* Manage information about each department.
* Associate employees with departments.
3. **Performance Management**:
* Track employee performance and status within the company.
* Managers can add performance feedback for employees who report to them.
* Employees can view their performance feedback and add personal feedback.
4. **Compensation Management**:
* HR can update employee compensation and receive AI-driven recommendations.
* Track compensation history for each employee.
5. **Benefits Management**:
* Track the benefits each employee receives.
6. **AI Integration**:
* AI provides recommendations for compensation adjustments based on trends and employee performance.
* Co-pilot assistance for managers to generate personalized performance feedback.
7. **Reporting and Summarization**:
* Summarize employee information, performance trends, and compensation data.
* Managers can view information about employees who report to them.
### Postgres Schemas
### 1. departments
This table stores information about each department, including its name and description, and is linked to a specific tenant.
```sql theme={null}
CREATE TABLE departments (
tenant_id UUID,
department_id UUID DEFAULT gen_random_uuid(),
department_name VARCHAR(100) NOT NULL,
description TEXT,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, department_id),
FOREIGN KEY (tenant_id) REFERENCES tenants(id)
);
```
### 2. employees
This table stores detailed information about employees, including their title, department, compensation, status, hire date, and termination date. It ensures each employee is associated with a specific tenant and department.
```sql theme={null}
CREATE TABLE employees (
tenant_id UUID,
employee_id UUID DEFAULT gen_random_uuid(),
department_id UUID,
title VARCHAR(100),
compensation DECIMAL(10, 2),
status VARCHAR(50),
hire_date DATE,
termination_date DATE,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, employee_id),
FOREIGN KEY (tenant_id, employee_id) REFERENCES users.tenant_users(tenant_id, user_id),
FOREIGN KEY (tenant_id, department_id) REFERENCES departments(tenant_id, department_id)
);
```
### 3. performance
This table tracks performance feedback for employees, including feedback text, rating, and the manager who provided the feedback. It is linked to the employee and the manager providing the feedback. The vector embeddings are calculated for each performance of an employee. This helps HR and managers to search about the feedback and even retrieve employees with similar feedback and come up with a cohesive plan for improvement.
```sql theme={null}
CREATE TABLE performance (
tenant_id UUID,
performance_id UUID DEFAULT gen_random_uuid(),
employee_id UUID,
manager_id UUID,
feedback TEXT,
rating INT,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, employee_id),
FOREIGN KEY (tenant_id, employee_id) REFERENCES employees(tenant_id, employee_id),
FOREIGN KEY (tenant_id, manager_id) REFERENCES employees(tenant_id, employee_id)
);
```
### 4. compensation\_history
This table records changes in employee compensation, including the old and new compensation amounts, the date of change, and the reason for the change. It helps track the history of compensation adjustments.
```sql theme={null}
CREATE TABLE compensation_history (
tenant_id UUID,
compensation_id UUID DEFAULT gen_random_uuid(),
employee_id UUID,
old_compensation DECIMAL(10, 2),
new_compensation DECIMAL(10, 2),
change_date DATE,
reason TEXT,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, employee_id),
FOREIGN KEY (tenant_id, employee_id) REFERENCES employees(tenant_id, employee_id)
);
```
### 5. employee\_feedback
This table stores personal feedback provided by employees. It ensures feedback is linked to the specific employee who provided it. The embeddings on all the feedback of an employee helps to track similar patterns across feedbacks. This can help managers to understand which is the top priority improvement for the employee.
```sql theme={null}
CREATE TABLE employee_feedback (
tenant_id UUID,
feedback_id UUID DEFAULT gen_random_uuid(),
employee_id UUID,
feedback TEXT,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, employee_id),
FOREIGN KEY (tenant_id, employee_id) REFERENCES employees(tenant_id, employee_id)
);
```
### 6. benefits
This table tracks the benefits each employee receives, including the name and description of the benefit. The embeddings are useful on the benefit types. This helps to summarize the long documents that explain benefits to the employees.
```sql theme={null}
CREATE TABLE benefits (
tenant_id UUID,
benefit_id UUID DEFAULT gen_random_uuid(),
employee_id UUID,
benefit_name VARCHAR(100),
benefit_description TEXT,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, employee_id),
FOREIGN KEY (tenant_id, employee_id) REFERENCES employees(tenant_id, employee_id)
);
```
### Full Script
This script includes all necessary tables for managing employees, performance, compensation history, feedback, departments, and benefits, following the multitenant rules with UUID types and primary and foreign keys defined appropriately.
```sql theme={null}
CREATE TABLE departments (
tenant_id UUID,
department_id UUID DEFAULT gen_random_uuid(),
department_name VARCHAR(100) NOT NULL,
description TEXT,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, department_id),
FOREIGN KEY (tenant_id) REFERENCES tenants(id)
);
CREATE TABLE employees (
tenant_id UUID,
employee_id UUID DEFAULT gen_random_uuid(),
department_id UUID,
title VARCHAR(100),
compensation DECIMAL(10, 2),
status VARCHAR(50),
hire_date DATE,
termination_date DATE,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, employee_id),
FOREIGN KEY (tenant_id, employee_id) REFERENCES users.tenant_users(tenant_id, user_id),
FOREIGN KEY (tenant_id, department_id) REFERENCES departments(tenant_id, department_id)
);
CREATE TABLE performance (
tenant_id UUID,
performance_id UUID DEFAULT gen_random_uuid(),
employee_id UUID,
manager_id UUID,
feedback TEXT,
rating INT,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, employee_id),
FOREIGN KEY (tenant_id, employee_id) REFERENCES employees(tenant_id, employee_id),
FOREIGN KEY (tenant_id, manager_id) REFERENCES employees(tenant_id, employee_id)
);
CREATE TABLE compensation_history (
tenant_id UUID,
compensation_id UUID DEFAULT gen_random_uuid(),
employee_id UUID,
old_compensation DECIMAL(10, 2),
new_compensation DECIMAL(10, 2),
change_date DATE,
reason TEXT,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, employee_id),
FOREIGN KEY (tenant_id, employee_id) REFERENCES employees(tenant_id, employee_id)
);
CREATE TABLE employee_feedback (
tenant_id UUID,
feedback_id UUID DEFAULT gen_random_uuid(),
employee_id UUID,
feedback TEXT,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, employee_id),
FOREIGN KEY (tenant_id, employee_id) REFERENCES employees(tenant_id, employee_id)
);
CREATE TABLE benefits (
tenant_id UUID,
benefit_id UUID DEFAULT gen_random_uuid(),
employee_id UUID,
benefit_name VARCHAR(100),
benefit_description TEXT,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, employee_id),
FOREIGN KEY (tenant_id, employee_id) REFERENCES employees(tenant_id, employee_id)
);
```
# TaskPilot AI - Your AI-Driven Guide to Seamless Task Management
Source: https://thenile.dev/docs/getting-started/usecases/issue-tracking
This application allows users to manage tasks, track status, search past tasks, and view task descriptions. It includes advanced AI features, such as a co-pilot that assists in adding tasks, asking questions about past tasks, and analyzing task execution speed. Users can also search for previous tasks. The application supports project management, with tasks organized under projects and a global roadmap for planning tasks across projects.
### Detailed Requirements
1. **Multitenancy:** Each tenant has its own isolated data, ensuring data security and segregation. The tenants can be placed close to the customer for better latency and needs to satisfy compliance requirements of the customers.
2. **Task Management:** Ability to create, update, and delete tasks. Each task has a name, description, status, and due date.
3. **Project Management:** Tasks are organized under projects. Each project can have multiple tasks.
4. **Task Status Tracking:** History of status changes for each task is tracked.
5. **Task Comments:** Users can add comments to tasks.
6. **Global Planning:** A roadmap that includes tasks from across projects for global planning.
7. **AI Features:**
* Co-pilot for task creation and querying past tasks.
* Insights on task execution speed and best practices for task management.
8. **User Management:** Users can belong to multiple tenants, with roles assigned to users per tenant.
9. **Search Functionality:** Ability to search for past tasks.
### Postgres Schemas
### 1. projects
Stores information about each project. The application can support multiple projects.
```sql theme={null}
CREATE TABLE projects (
tenant_id UUID,
project_id UUID DEFAULT gen_random_uuid(),
user_id UUID,
project_name VARCHAR(100) NOT NULL,
project_description TEXT,
start_date DATE,
end_date DATE,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, project_id),
FOREIGN KEY (tenant_id) REFERENCES tenants(tenant_id),
FOREIGN KEY (user_id) REFERENCES users(user_id)
);
```
### 2. tasks
Stores information about each task, linked to a project. Each project can have multiple tasks. For each task, we track vector embeddings of the description field. In a real world application, you may need to chunk the description of the tasks and have a table to track embeddings for each chunk. The model itself can be useful to figure the right way to chunk the description to ensure that the context is preserved.
```sql theme={null}
CREATE TABLE tasks (
tenant_id UUID,
task_id UUID DEFAULT gen_random_uuid(),
project_id UUID,
user_id UUID,
task_name VARCHAR(100) NOT NULL,
task_description TEXT,
task_status VARCHAR(50) NOT NULL,
due_date DATE,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, task_id),
FOREIGN KEY (tenant_id, project_id) REFERENCES projects(tenant_id, project_id),
FOREIGN KEY (tenant_id) REFERENCES tenants(tenant_id),
FOREIGN KEY (user_id) REFERENCES users(user_id)
);
```
### 3. task\_status\_history
Tracks the status changes for each task. The status of each task can also be converted to embeddings if they need to be searched using AI models.
```sql theme={null}
CREATE TABLE task_status_history (
tenant_id UUID,
history_id UUID DEFAULT gen_random_uuid(),
task_id UUID,
old_status VARCHAR(50) NOT NULL,
new_status VARCHAR(50) NOT NULL,
changed_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
changed_by UUID,
PRIMARY KEY (tenant_id, history_id),
FOREIGN KEY (tenant_id, task_id) REFERENCES tasks(tenant_id, task_id),
FOREIGN KEY (tenant_id, changed_by) REFERENCES tenant_users(tenant_id, user_id)
);
```
### 4. task\_comments
Stores comments added to each task. The embeddings are created for each comments. Typically, comments are going to be short under tasks and one embedding per comment should be a good chunking length.
```sql theme={null}
CREATE TABLE task_comments (
tenant_id UUID,
comment_id UUID DEFAULT gen_random_uuid(),
task_id UUID,
user_id UUID,
comment_text TEXT NOT NULL,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, comment_id),
FOREIGN KEY (tenant_id, task_id) REFERENCES tasks(tenant_id, task_id),
FOREIGN KEY (tenant_id, user_id) REFERENCES tenant_users(tenant_id, user_id)
);
```
### 5. roadmap
Contains tasks from across projects for global planning.
```sql theme={null}
CREATE TABLE roadmap (
tenant_id UUID,
roadmap_id UUID DEFAULT gen_random_uuid(),
task_id UUID,
user_id UUID,
milestone_date DATE,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, roadmap_id),
FOREIGN KEY (tenant_id, task_id) REFERENCES tasks(tenant_id, task_id),
FOREIGN KEY (tenant_id) REFERENCES tenants(tenant_id),
FOREIGN KEY (user_id) REFERENCES users(user_id)
);
```
### Complete SQL Script
```sql theme={null}
-- Create projects table
CREATE TABLE projects (
tenant_id UUID,
project_id UUID DEFAULT gen_random_uuid(),
user_id UUID,
project_name VARCHAR(100) NOT NULL,
project_description TEXT,
start_date DATE,
end_date DATE,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, project_id),
FOREIGN KEY (tenant_id, user_id) REFERENCES tenant_users(tenant_id, user_id)
);
-- Create tasks table linked to projects
CREATE TABLE tasks (
tenant_id UUID,
task_id UUID DEFAULT gen_random_uuid(),
project_id UUID,
user_id UUID,
task_name VARCHAR(100) NOT NULL,
task_description TEXT,
task_status VARCHAR(50) NOT NULL,
due_date DATE,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, task_id),
FOREIGN KEY (tenant_id, project_id) REFERENCES projects(tenant_id, project_id),
FOREIGN KEY (tenant_id, user_id) REFERENCES tenant_users(tenant_id, user_id)
);
-- Create task_status_history table
CREATE TABLE task_status_history (
tenant_id UUID,
history_id UUID DEFAULT gen_random_uuid(),
task_id UUID,
old_status VARCHAR(50) NOT NULL,
new_status VARCHAR(50) NOT NULL,
changed_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
changed_by UUID,
PRIMARY KEY (tenant_id, history_id),
FOREIGN KEY (tenant_id, task_id) REFERENCES tasks(tenant_id, task_id),
FOREIGN KEY (tenant_id, changed_by) REFERENCES tenant_users(tenant_id, user_id)
);
-- Create task_comments table
CREATE TABLE task_comments (
tenant_id UUID,
comment_id UUID DEFAULT gen_random_uuid(),
task_id UUID,
user_id UUID,
comment_text TEXT NOT NULL,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, comment_id),
FOREIGN KEY (tenant_id, task_id) REFERENCES tasks(tenant_id, task_id),
FOREIGN KEY (tenant_id, user_id) REFERENCES tenant_users(tenant_id, user_id)
);
-- Create roadmap table for global planning
CREATE TABLE roadmap (
tenant_id UUID,
roadmap_id UUID DEFAULT gen_random_uuid(),
task_id UUID,
user_id UUID,
milestone_date DATE,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, roadmap_id),
FOREIGN KEY (tenant_id, task_id) REFERENCES tasks(tenant_id, task_id),
FOREIGN KEY (tenant_id, user_id) REFERENCES tenant_users(tenant_id, user_id)
);
```
This script sets up the necessary tables and relationships to support a robust and scalable multitenant task management application with advanced AI capabilities, including project management, task tracking, task status history, task comments, and global planning through a roadmap.
# SalesLeadPilot - AI-Driven Guidance for Superior Lead Management
Source: https://thenile.dev/docs/getting-started/usecases/lead-management
The Sales Lead Management application is designed to allow multiple organizations (tenants) to manage their sales leads efficiently. This application includes advanced AI features to assist in adding leads, querying past leads, and suggesting future actions to increase conversion rates. Key functionalities include lead management, status tracking, lead quality assessment, conversation tracking, note-taking, and comprehensive search capabilities.
1. **Lead Management:** Create, update, and manage sales leads.
2. **Status Tracking:** Track the status of each lead (e.g., new, contacted, qualified, unqualified, closed).
3. **Lead Quality:** Mark the quality of leads (e.g., hot, warm, cold).
4. **Search Leads:** Search for past leads based on various criteria.
5. **Conversations and Notes:** Track conversations and take notes for each lead.
6. **AI Co-Pilot:** Assist in adding leads, asking questions about past leads, and suggesting best steps to take for future conversions.
7. **Search with AI:** Utilize AI to search previous leads and provide insights.
### Postgres Schemas
### 1. leads
Stores information about each sales lead. A sales person finds the right lead for a company and adds them to the system. Each lead goes through a pipeline from email reach out, phone conversation to deal closing or disqualified.
```sql theme={null}
CREATE TABLE leads (
tenant_id UUID,
lead_id UUID DEFAULT gen_random_uuid(),
user_id UUID,
lead_name VARCHAR(100) NOT NULL,
lead_email VARCHAR(100) NOT NULL,
lead_phone VARCHAR(20),
lead_status VARCHAR(50) NOT NULL,
lead_quality VARCHAR(50),
deal_size DECIMAL(10, 2),
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, lead_id),
FOREIGN KEY (tenant_id, user_id) REFERENCES tenant_users(tenant_id, user_id)
);
```
### 2. lead\_notes
Stores notes related to each lead. These are notes taken by the sales rep as they go through the process of converting a lead to a potential customer. The embeddings of these notes are stored in the table to help RAG to the notes.
```sql theme={null}
CREATE TABLE lead_notes (
tenant_id UUID,
note_id UUID DEFAULT gen_random_uuid(),
lead_id UUID,
user_id UUID,
note_content TEXT NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, note_id),
FOREIGN KEY (tenant_id, lead_id) REFERENCES leads(tenant_id, lead_id),
FOREIGN KEY (tenant_id, user_id) REFERENCES tenant_users(tenant_id, user_id)
);
```
### 3. lead\_conversations
Stores conversations related to each lead, with AI-generated summaries and vector embeddings for advanced search and analysis. There are multiple things to note here. The embeddings are stored per lead conversation here. These conversations can be long and having a single embedding may not be ideal. You may want to chunk them to get better contextual search.
```sql theme={null}
CREATE TABLE lead_conversations (
tenant_id UUID,
conversation_id UUID DEFAULT gen_random_uuid(),
lead_id UUID,
user_id UUID,
conversation_content TEXT NOT NULL,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, conversation_id),
FOREIGN KEY (tenant_id, lead_id) REFERENCES leads(tenant_id, lead_id),
FOREIGN KEY (tenant_id, user_id) REFERENCES tenant_users(tenant_id, user_id)
);
```
### 4. lead\_status\_history
Tracks the status changes for each lead.
```sql theme={null}
CREATE TABLE lead_status_history (
tenant_id UUID,
history_id UUID DEFAULT gen_random_uuid(),
lead_id UUID,
old_status VARCHAR(50) NOT NULL,
new_status VARCHAR(50) NOT NULL,
changed_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
changed_by UUID,
PRIMARY KEY (tenant_id, history_id),
FOREIGN KEY (tenant_id, lead_id) REFERENCES leads(tenant_id, lead_id),
FOREIGN KEY (tenant_id, changed_by) REFERENCES tenant_users(tenant_id, user_id)
);
```
### 5. global\_companies
Tracks the top 1500 companies in the world. This table is shared across all tenants. This is not a per tenant table. All the sales reps in all the tenants/customers can share this data.
```sql theme={null}
CREATE TABLE global_companies (
company_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
company_name VARCHAR(100) NOT NULL,
industry VARCHAR(100),
country VARCHAR(50),
revenue DECIMAL(15, 2),
employees INTEGER,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
```
### Complete SQL Script
```sql theme={null}
-- Create leads table
CREATE TABLE leads (
tenant_id UUID,
lead_id UUID DEFAULT gen_random_uuid(),
user_id UUID,
lead_name VARCHAR(100) NOT NULL,
lead_email VARCHAR(100) NOT NULL,
lead_phone VARCHAR(20),
lead_status VARCHAR(50) NOT NULL,
lead_quality VARCHAR(50),
deal_size DECIMAL(10, 2),
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, lead_id),
FOREIGN KEY (tenant_id, user_id) REFERENCES tenant_users(tenant_id, user_id)
);
-- Create lead_notes table
CREATE TABLE lead_notes (
tenant_id UUID,
note_id UUID DEFAULT gen_random_uuid(),
lead_id UUID,
user_id UUID,
note_content TEXT NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, note_id),
FOREIGN KEY (tenant_id, lead_id) REFERENCES leads(tenant_id, lead_id),
FOREIGN KEY (tenant_id, user_id) REFERENCES tenant_users(tenant_id, user_id)
);
-- Create lead_conversations table with AI-generated summaries and vector embeddings
CREATE TABLE lead_conversations (
tenant_id UUID,
conversation_id UUID DEFAULT gen_random_uuid(),
lead_id UUID,
user_id UUID,
conversation_content TEXT NOT NULL,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, conversation_id),
FOREIGN KEY (tenant_id, lead_id) REFERENCES leads(tenant_id, lead_id),
FOREIGN KEY (tenant_id, user_id) REFERENCES tenant_users(tenant_id, user_id)
);
-- Create lead_status_history table
CREATE TABLE lead_status_history (
tenant_id UUID,
history_id UUID DEFAULT gen_random_uuid(),
lead_id UUID,
old_status VARCHAR(50) NOT NULL,
new_status VARCHAR(50) NOT NULL,
changed_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
changed_by UUID,
PRIMARY KEY (tenant_id, history_id),
FOREIGN KEY (tenant_id, lead_id) REFERENCES leads(tenant_id, lead_id),
FOREIGN KEY (tenant_id, changed_by) REFERENCES tenant_users(tenant_id, user_id)
);
-- Create global_companies table
CREATE TABLE global_companies (
company_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
company_name VARCHAR(100) NOT NULL,
industry VARCHAR(100),
country VARCHAR(50),
revenue DECIMAL(15, 2),
employees INTEGER,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
```
This script sets up the necessary tables and relationships for a comprehensive multitenant sales lead management application with advanced AI capabilities.
# LawPilot AI - AI That Thinks Like a Lawyer
Source: https://thenile.dev/docs/getting-started/usecases/legal
This application will assist law firms in managing legal documents, streamline the document creation and editing process, and facilitate client interaction. Lawyers can create, edit, and chat with legal documents. Clients can read, sign, and chat with the documents. AI features include summarizing legal documents and parsing uploaded agreements for searchability.
### Detailed Requirements
1. **Multitenancy:** Each law firm (tenant) has its own isolated data to ensure data security and segregation.
2. **User Management:** Lawyers and clients can belong to multiple tenants.
3. **Document Management:**
* Lawyers can create and edit legal documents.
* Lawyers and clients can chat within the document.
* Documents can be shared with clients.
4. **Client Interaction:**
* Clients can read, sign, and chat with legal documents.
5. **Case Management:** Track cases within the firm, assign cases to lawyers and clients, and maintain case details.
6. **Contract Management:** Track legal contracts associated with cases, including the ability to upload, create, and summarize contracts.
7. **Contract Status:** Track the status of contracts (created, read, signed, revoked).
### Postgres Schemas
### 1. lawyers
Tracks information about lawyers, including their education and experience. The lawyers are users of the system and belong to a specific tenant. They have to be one of the users registered in the system.
```sql theme={null}
CREATE TABLE lawyers (
tenant_id UUID,
lawyer_id UUID,
education TEXT,
experience TEXT,
PRIMARY KEY (tenant_id, lawyer_id),
FOREIGN KEY (tenant_id, lawyer_id) REFERENCES users.tenant_users(tenant_id, user_id)
);
```
### 2. clients
Tracks information about clients, including their personal information. The clients are users of the system and belong to a specific tenant. They have to be one of the users registered in the system.
```sql theme={null}
CREATE TABLE clients (
tenant_id UUID,
client_id UUID,
personal_info JSONB,
PRIMARY KEY (tenant_id, client_id),
FOREIGN KEY (tenant_id, client_id) REFERENCES users.tenant_users(tenant_id, user_id)
);
```
### 3. cases
Tracks all cases within the firm, including assigned tenant, lawyer, and client. The cases have detailed description about the case and both lawyers and clients would want to ask questions about it. The vector embeddings are created for each of the case descriptions that are used to implement a RAG architecture to enable search capabilities. The AI can also study new cases based on past cases and propose next steps.
```sql theme={null}
CREATE TABLE cases (
tenant_id UUID,
case_id UUID DEFAULT gen_random_uuid(),
lawyer_id UUID,
client_id UUID,
case_description TEXT,
status VARCHAR(50),
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
PRIMARY KEY (tenant_id, case_id),
FOREIGN KEY (tenant_id, lawyer_id) REFERENCES lawyers(tenant_id, lawyer_id),
FOREIGN KEY (tenant_id, client_id) REFERENCES clients(tenant_id, client_id)
);
```
### 4. contracts
Tracks all legal contracts relevant to a case. Contracts can be uploaded or created. These contracts are typically really long and would be a great use case to help summarize and also ask questions. The embeddings are calculated for each of the contracts. For simplicity, this schema calculates one embedding per contract. In a real world application, you would want to create an embedding per contract chunk. This means breaking the contract into chunks and creating a table that tracks embeddings per chunk. These chunks can then be fed into the AI model for Q\&A or summarization.
```sql theme={null}
CREATE TABLE contracts (
tenant_id UUID,
contract_id UUID DEFAULT gen_random_uuid(),
case_id UUID,
contract_content TEXT,
created_by UUID,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
PRIMARY KEY (tenant_id, contract_id),
FOREIGN KEY (tenant_id, case_id) REFERENCES cases(tenant_id, case_id),
FOREIGN KEY (tenant_id, created_by) REFERENCES lawyers(tenant_id, lawyer_id)
);
```
### 5. contract\_status
Tracks the status of contracts (created, read, signed, revoked).
```sql theme={null}
CREATE TABLE contract_status (
tenant_id UUID,
status_id UUID DEFAULT gen_random_uuid(),
contract_id UUID,
status VARCHAR(50),
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, status_id),
FOREIGN KEY (tenant_id, contract_id) REFERENCES contracts(tenant_id, contract_id)
);
```
### Complete SQL Script
```sql theme={null}
-- Create lawyers table
CREATE TABLE lawyers (
tenant_id UUID,
lawyer_id UUID,
education TEXT,
experience TEXT,
PRIMARY KEY (tenant_id, lawyer_id),
FOREIGN KEY (tenant_id, lawyer_id) REFERENCES users.tenant_users(tenant_id, user_id)
);
-- Create clients table
CREATE TABLE clients (
tenant_id UUID,
client_id UUID,
personal_info JSONB,
PRIMARY KEY (tenant_id, client_id),
FOREIGN KEY (tenant_id, client_id) REFERENCES users.tenant_users(tenant_id, user_id)
);
-- Create cases table
CREATE TABLE cases (
tenant_id UUID,
case_id UUID DEFAULT gen_random_uuid(),
lawyer_id UUID,
client_id UUID,
case_description TEXT,
status VARCHAR(50),
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
PRIMARY KEY (tenant_id, case_id),
FOREIGN KEY (tenant_id, lawyer_id) REFERENCES lawyers(tenant_id, lawyer_id),
FOREIGN KEY (tenant_id, client_id) REFERENCES clients(tenant_id, client_id)
);
-- Create contracts table
CREATE TABLE contracts (
tenant_id UUID,
contract_id UUID DEFAULT gen_random_uuid(),
case_id UUID,
contract_content TEXT,
created_by UUID,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
PRIMARY KEY (tenant_id, contract_id),
FOREIGN KEY (tenant_id, case_id) REFERENCES cases(tenant_id, case_id),
FOREIGN KEY (tenant_id, created_by) REFERENCES lawyers(tenant_id, lawyer_id)
);
-- Create contract_status table
CREATE TABLE contract_status (
tenant_id UUID,
status_id UUID DEFAULT gen_random_uuid(),
contract_id UUID,
status VARCHAR(50),
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, status_id),
FOREIGN KEY (tenant_id, contract_id) REFERENCES contracts(tenant_id, contract_id)
);
```
# SmartCampaign AI - Your AI-Driven Marketing Ally
Source: https://thenile.dev/docs/getting-started/usecases/marketing
The Marketing Campaigns Application is a comprehensive platform designed to streamline and enhance the process of managing marketing campaigns. It supports users in creating and managing campaigns, tracking performance, and leveraging AI to optimize marketing efforts. The system allows for detailed contact management, effective campaign creation, and in-depth analytics, while providing intelligent insights to improve conversion rates and campaign effectiveness.
### Key Features:
1. **Contact Management:**
* **Add Contacts:** Users can manually or programmatically add contacts to the system.
* **Contact Information:** Store detailed information about each contact, including:
* First Name
* Last Name
* Email
* Phone Number
* Descriptive information (notes about the contact)
2. **Campaign Creation and Management:**
* **Create Campaigns:** Users can create new marketing campaigns.
* **Associate Contacts:** Each campaign can have a list of associated contacts.
* **Email Template:** Attach an email template to each campaign, which will be used to send out the emails.
* **Campaign Status:** Track the status of each campaign (e.g., draft, sent, completed).
3. **Campaign Analytics:**
* **Email Sent:** Track the number of emails sent out for each campaign.
* **Email Opened:** Track the number of emails that were opened by the recipients.
* **Links Clicked:** Track the number of clicks on links within the email.
* **Total Clicks:** Track the total number of overall clicks on the email.
4. **AI Features:**
* **Increase Conversion Rates:** Use AI to provide insights and suggestions on how to increase conversion rates.
* **Search Contacts:** AI can search through contacts to find the ideal set of contacts for a specific campaign.
* **Campaign Insights:** AI can search past campaigns to get insights and suggest new campaigns to create based on previous performance.
* **Drafting Emails:** AI can help draft personalized emails for each contact within a campaign.
* **Summarize Campaign Performance:** AI can summarize the performance of a campaign based on various metrics.
### Postgres Schemas
### 1. contacts
This table stores detailed information about contacts, including their email addresses, names, phone numbers, and additional descriptive notes. It helps in managing the contact base for marketing campaigns and ensures each contact is associated with a specific tenant.
```sql theme={null}
CREATE TABLE contacts (
tenant_id UUID,
contact_id UUID DEFAULT gen_random_uuid(),
email VARCHAR(255) NOT NULL,
first_name VARCHAR(100),
last_name VARCHAR(100),
phone_number VARCHAR(20),
description TEXT,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, contact_id),
FOREIGN KEY (tenant_id) REFERENCES tenants(id)
);
```
### 2. campaigns
This table holds information about marketing campaigns, including the campaign's name, description, start date, and current status. It enables the management of various campaigns and associates each with a tenant. We track the embeddings for all the campaigns. The embeddings are generated from the campaign description. This helps the RAG architecture to make the AI propose similar campaigns that can be executed in the future and search the past campaigns.
```sql theme={null}
CREATE TABLE campaigns (
tenant_id UUID,
campaign_id UUID DEFAULT gen_random_uuid(),
name VARCHAR(255) NOT NULL,
description TEXT,
start_date DATE,
status VARCHAR(50) DEFAULT 'draft',
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
PRIMARY KEY (tenant_id, campaign_id),
FOREIGN KEY (tenant_id) REFERENCES tenants(id)
);
```
### 3. campaign\_contacts
This table maps contacts to campaigns, indicating which contacts are included in which campaigns. It supports many-to-many relationships between contacts and campaigns, allowing detailed campaign targeting.
```sql theme={null}
CREATE TABLE campaign_contacts (
tenant_id UUID,
campaign_id UUID,
contact_id UUID,
PRIMARY KEY (tenant_id, campaign_id, contact_id),
FOREIGN KEY (tenant_id, campaign_id) REFERENCES campaigns(tenant_id, campaign_id),
FOREIGN KEY (tenant_id, contact_id) REFERENCES contacts(tenant_id, contact_id)
);
```
### 4. campaign\_emails
This table records the details of the emails sent for each campaign, including the subject, body, and sent date. It helps track email communications and their association with specific campaigns. The embeddings are calculated from the email content. These are useful for the AI to draft future emails that had much better conversion, summarize emails and also help search through the past emails across campaigns.
```sql theme={null}
CREATE TABLE campaign_emails (
tenant_id UUID,
campaign_id UUID,
email_id UUID DEFAULT gen_random_uuid(),
subject VARCHAR(255),
body TEXT,
sent_date TIMESTAMP,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
PRIMARY KEY (tenant_id, campaign_id),
FOREIGN KEY (tenant_id, campaign_id) REFERENCES campaigns(tenant_id, campaign_id)
);
```
### 5. campaign\_analytics
This table tracks the performance metrics of each campaign, such as the number of emails sent, opened, links clicked, and conversions achieved. It provides insights into campaign effectiveness and is associated with a specific tenant and campaign.
```sql theme={null}
CREATE TABLE campaign_analytics (
tenant_id UUID,
campaign_id UUID,
analytics_id UUID DEFAULT gen_random_uuid(),
email_sent_count INT DEFAULT 0,
email_opened_count INT DEFAULT 0,
link_clicked_count INT DEFAULT 0,
conversions INT DEFAULT 0,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, campaign_id, analytics_id),
FOREIGN KEY (tenant_id, campaign_id) REFERENCES campaigns(tenant_id, campaign_id)
);
```
### Complete SQL Script
```sql theme={null}
-- Table for storing contacts information
CREATE TABLE contacts (
tenant_id UUID,
contact_id UUID DEFAULT gen_random_uuid(),
email VARCHAR(255) NOT NULL,
first_name VARCHAR(100),
last_name VARCHAR(100),
phone_number VARCHAR(20),
description TEXT,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, contact_id),
FOREIGN KEY (tenant_id) REFERENCES tenants(id)
);
-- Table for storing campaign information
CREATE TABLE campaigns (
tenant_id UUID,
campaign_id UUID DEFAULT gen_random_uuid(),
name VARCHAR(255) NOT NULL,
description TEXT,
start_date DATE,
status VARCHAR(50) DEFAULT 'draft',
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
PRIMARY KEY (tenant_id, campaign_id),
FOREIGN KEY (tenant_id) REFERENCES tenants(id)
);
-- Table for mapping contacts to campaigns
CREATE TABLE campaign_contacts (
tenant_id UUID,
campaign_id UUID,
contact_id UUID,
PRIMARY KEY (tenant_id, campaign_id, contact_id),
FOREIGN KEY (tenant_id, campaign_id) REFERENCES campaigns(tenant_id, campaign_id),
FOREIGN KEY (tenant_id, contact_id) REFERENCES contacts(tenant_id, contact_id)
);
-- Table for storing information about campaign emails
CREATE TABLE campaign_emails (
tenant_id UUID,
campaign_id UUID,
email_id UUID DEFAULT gen_random_uuid(),
subject VARCHAR(255),
body TEXT,
sent_date TIMESTAMP,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
PRIMARY KEY (tenant_id, campaign_id),
FOREIGN KEY (tenant_id, campaign_id) REFERENCES campaigns(tenant_id, campaign_id)
);
-- Table for storing campaign performance analytics
CREATE TABLE campaign_analytics (
tenant_id UUID,
campaign_id UUID,
analytics_id UUID DEFAULT gen_random_uuid(),
email_sent_count INT DEFAULT 0,
email_opened_count INT DEFAULT 0,
link_clicked_count INT DEFAULT 0,
conversions INT DEFAULT 0,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, campaign_id, analytics_id),
FOREIGN KEY (tenant_id, campaign_id) REFERENCES campaigns(tenant_id, campaign_id)
);
```
# SmartNotion AI - Crafting Organized Thoughts with Artificial Intelligence
Source: https://thenile.dev/docs/getting-started/usecases/notion
SmartNotion AI allows tenants to create, edit, manage, and search documents within their workspace. The application incorporates AI features for editing, summarizing, and searching documents, including text and images. Additionally, documents should be stored close to the customer for compliance reasons, and the document editing experience must be very fast.
### Key Features:
1. **Document Creation and Management**:
* Create and manage documents within a tenant's workspace.
* Support for rich text editing and multi-modal content (text and images).
* Track document metadata such as creation date, last modified date, and author.
* Ensure documents are stored close to the customer for compliance purposes.
2. **AI-Powered Editing and Summarization**:
* Use AI to assist with editing and modifying documents.
* AI-driven document summarization to provide quick overviews of content.
3. **Advanced Search Functionality**:
* Search across documents within a tenant's workspace.
* AI-driven search to find text and images within documents.
* Show similar documents based on content and context.
4. **Document Collaboration**:
* Support for collaborative editing and version control.
* Track changes and comments from multiple users.
5. **Security and Permissions**:
* Role-based access control to manage who can view, edit, and delete documents.
* Ensure document data is securely stored and managed within the tenant's workspace.
6. **Performance**:
* Ensure a fast and responsive document editing experience.
### Postgres Schemas
### 1. documents
Stores document information including title, content, creation and modification timestamps, author, and storage location. The embeddings for the documents are stored in a separate table to ensure these rows are not too long which can cause performance impact.
```sql theme={null}
CREATE TABLE documents (
tenant_id UUID,
document_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
title VARCHAR(255),
content TEXT,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
author_id UUID,
storage_location VARCHAR(255), -- Location where the document is stored
FOREIGN KEY (tenant_id) REFERENCES tenants(tenant_id),
FOREIGN KEY (tenant_id, author_id) REFERENCES tenant_users(tenant_id, user_id)
);
```
### 2. document\_versions
Tracks versions of documents, storing historical content and metadata for each version. The old version help users to browse the history of a document and even ask AI questions on how and who made changes to the document.
```sql theme={null}
CREATE TABLE document_versions (
tenant_id UUID,
document_id UUID,
version_id UUID DEFAULT gen_random_uuid(),
version_number INT,
content TEXT,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
author_id UUID,
PRIMARY KEY (tenant_id, document_id, version_id),
FOREIGN KEY (tenant_id, document_id) REFERENCES documents(tenant_id, document_id),
FOREIGN KEY (tenant_id, author_id) REFERENCES users.tenant_users(tenant_id, user_id)
);
```
### 3. document\_comments
Stores comments made on documents, linking comments to specific documents and users. Each comment also stores its embeddings along with it. This helps to understand if a document has received generally positive or negative comments using AI. It can also help to summarize all the comments and even search through them.
```sql theme={null}
CREATE TABLE document_comments (
tenant_id UUID,
comment_id UUID DEFAULT gen_random_uuid(),
document_id UUID,
user_id UUID,
comment TEXT,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, document_id),
FOREIGN KEY (tenant_id, document_id) REFERENCES documents(tenant_id, document_id),
FOREIGN KEY (tenant_id, user_id) REFERENCES users.tenant_users(tenant_id, user_id)
);
```
### 4. teamspaces
Tracks teamspaces created within tenants, storing metadata such as name, description, and creator.
```sql theme={null}
CREATE TABLE teamspaces (
tenant_id UUID,
teamspace_id UUID DEFAULT gen_random_uuid(),
name VARCHAR(255),
description TEXT,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
created_by UUID,
PRIMARY KEY (tenant_id, teamspace_id),
FOREIGN KEY (tenant_id, created_by) REFERENCES users.tenant_users(tenant_id, user_id)
);
```
### 5. teamspace\_documents
Maps documents to teamspaces, establishing relationships between teamspaces and their documents.
```sql theme={null}
CREATE TABLE teamspace_documents (
teamspace_id UUID,
tenant_id UUID,
document_id UUID,
PRIMARY KEY (tenant_id, teamspace_id, document_id),
FOREIGN KEY (tenant_id, teamspace_id) REFERENCES teamspaces(tenant_id, teamspace_id),
FOREIGN KEY (tenant_id, document_id) REFERENCES documents(tenant_id, document_id)
);
```
### 6. document\_embeddings
Tracks vector embeddings for chunks of documents, supporting advanced AI-powered search capabilities. The chunking size is complex and not straightforward based on a size. It needs to be done based on context. The AI model itself can be used for this purpose to figure out what are the relevant parts of the document that can be chunked. A RAG architecture can then use these embeddings to summarize, search and even correct parts of the document.
```sql theme={null}
CREATE TABLE document_embeddings (
tenant_id UUID,
document_id UUID,
chunk_id UUID DEFAULT gen_random_uuid(),
chunk_content TEXT,
vector_embedding VECTOR,
PRIMARY KEY (tenant_id, document_id, chunk_id),
FOREIGN KEY (tenant_id, document_id) REFERENCES documents(tenant_id, document_id)
);
```
### 7. favourites
Tracks documents that users have marked as favorites within their tenant's workspace.
```sql theme={null}
CREATE TABLE favourites (
tenant_id UUID,
user_id UUID,
document_id UUID,
PRIMARY KEY (tenant_id, user_id, document_id),
FOREIGN KEY (tenant_id, user_id) REFERENCES users.tenant_users(tenant_id, user_id),
FOREIGN KEY (tenant_id, document_id) REFERENCES documents(tenant_id, document_id)
);
```
### Full Script
```sql theme={null}
CREATE TABLE documents (
tenant_id UUID,
document_id UUID DEFAULT gen_random_uuid(),
title VARCHAR(255),
content TEXT,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
author_id UUID,
storage_location VARCHAR(255), -- Location where the document is stored
PRIMARY KEY (tenant_id, document_id),
FOREIGN KEY (tenant_id) REFERENCES tenants(id),
FOREIGN KEY (tenant_id, author_id) REFERENCES users.tenant_users(tenant_id, user_id)
);
CREATE TABLE document_versions (
tenant_id UUID,
document_id UUID,
version_id UUID DEFAULT gen_random_uuid(),
version_number INT,
content TEXT,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
author_id UUID,
PRIMARY KEY (tenant_id, document_id, version_id),
FOREIGN KEY (tenant_id, document_id) REFERENCES documents(tenant_id, document_id),
FOREIGN KEY (tenant_id, author_id) REFERENCES users.tenant_users(tenant_id, user_id)
);
CREATE TABLE document_comments (
tenant_id UUID,
comment_id UUID DEFAULT gen_random_uuid(),
document_id UUID,
user_id UUID,
comment TEXT,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, document_id),
FOREIGN KEY (tenant_id, document_id) REFERENCES documents(tenant_id, document_id),
FOREIGN KEY (tenant_id, user_id) REFERENCES users.tenant_users(tenant_id, user_id)
);
CREATE TABLE teamspaces (
tenant_id UUID,
teamspace_id UUID DEFAULT gen_random_uuid(),
name VARCHAR(255),
description TEXT,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
created_by UUID,
PRIMARY KEY (tenant_id, teamspace_id),
FOREIGN KEY (tenant_id, created_by) REFERENCES users.tenant_users(tenant_id, user_id)
);
CREATE TABLE teamspace_documents (
teamspace_id UUID,
tenant_id UUID,
document_id UUID,
PRIMARY KEY (tenant_id, teamspace_id, document_id),
FOREIGN KEY (tenant_id, teamspace_id) REFERENCES teamspaces(tenant_id, teamspace_id),
FOREIGN KEY (tenant_id, document_id) REFERENCES documents(tenant_id, document_id)
);
CREATE TABLE document_embeddings (
tenant_id UUID,
document_id UUID,
chunk_id UUID DEFAULT gen_random_uuid(),
chunk_content TEXT,
vector_embedding VECTOR,
PRIMARY KEY (tenant_id, document_id, chunk_id),
FOREIGN KEY (tenant_id, document_id) REFERENCES documents(tenant_id, document_id)
);
CREATE TABLE favourites (
tenant_id UUID,
user_id UUID,
document_id UUID,
PRIMARY KEY (tenant_id, user_id, document_id),
FOREIGN KEY (tenant_id, user_id) REFERENCES users.tenant_users(tenant_id, user_id),
FOREIGN KEY (tenant_id, document_id) REFERENCES documents(tenant_id, document_id)
);
```
# OnboardIQ - Transform Customer Onboarding with Insightful AI
Source: https://thenile.dev/docs/getting-started/usecases/onboarding
The customer onboarding application is designed to streamline and manage the onboarding process for new customers. The system tracks customer projects, assigns support members, manages tasks, and utilizes AI to facilitate both customer and support team interactions. The application also integrates a knowledge base and provides notifications for project deadlines.
**Key Features:**
1. **Customer and Project Management:**
* Each customer can have multiple onboarding projects.
* Each project tracks progress, deadlines, and status.
2. **Task Management:**
* Tasks are created and assigned to both customers and support members.
* Tasks include detailed information needed for completion.
* Task status is tracked, including whether it is done or not.
3. **Support Member Assignment:**
* Each customer is assigned a support member who helps with onboarding.
* The support member and customer share task lists.
4. **AI Integration:**
* Customers can use AI to ask questions and receive help with onboarding tasks.
* Support members can use AI to summarize customer actions, troubleshoot issues, and get help to unblock the customer.
5. **Internal Knowledge Base:**
* An internal knowledge base is available to assist with onboarding.
* The AI utilizes the knowledge base to provide support and guidance.
6. **Progress Tracking and Notifications:**
* Progress status is monitored for each project.
* Notifications are sent to sales leaders if a project is delayed beyond the deadline.
### Postgres Schemas
### **1. customers**
The `customers` table stores information about the customers being onboarded. Each customer belongs to a specific tenant (organization) and has a unique identifier. The table also includes contact information for each customer.
```sql theme={null}
CREATE TABLE customers (
tenant_id UUID, -- Identifier for the tenant (organization) to which the customer belongs
customer_id UUID, -- Unique identifier for the customer
name VARCHAR(100) NOT NULL, -- Name of the customer
contact_info JSONB, -- JSON object to store various contact details like email, phone number, etc.
PRIMARY KEY (tenant_id, customer_id), -- Composite primary key combining tenant_id and customer_id
FOREIGN KEY (tenant_id) REFERENCES tenants(tenant_id) -- Foreign key reference to the tenants table
);
```
### **2. projects**
The `projects` table tracks onboarding projects for each customer. It includes details about the project's status, progress, and deadline. Each project is associated with a specific customer and tenant. The embeddings on the projects description helps support to find similar projects done before and learn from them.
```sql theme={null}
CREATE TABLE projects (
tenant_id UUID, -- Identifier for the tenant (organization) to which the project belongs
project_id UUID, -- Unique identifier for the project
customer_id UUID, -- Identifier for the customer associated with the project
name VARCHAR(100) NOT NULL, -- Name of the project
description TEXT NOT NULL, -- Detailed description of the project
status VARCHAR(50) NOT NULL, -- Current status of the project (e.g., In Progress, Completed)
deadline TIMESTAMP, -- Deadline for project completion
progress NUMERIC(5, 2), -- Progress of the project as a percentage (0 to 100)
vector_embedding VECTOR(768), -- Embedding vector for AI-related operations (e.g., summarization)
PRIMARY KEY (tenant_id, project_id), -- Composite primary key combining tenant_id and project_id
FOREIGN KEY (tenant_id, customer_id) REFERENCES customers(tenant_id, customer_id) -- Foreign key reference to the customers table
);
```
### **3. support\_members**
The `support_members` table records information about support personnel assigned to assist customers. Each support member is linked to a specific tenant and is referenced through the tenant\_users table.
```sql theme={null}
CREATE TABLE support_members (
tenant_id UUID, -- Identifier for the tenant (organization) to which the support member belongs
support_member_id UUID, -- Unique identifier for the support member
PRIMARY KEY (tenant_id, support_member_id), -- Composite primary key combining tenant_id and support_member_id
FOREIGN KEY (tenant_id, support_member_id) REFERENCES tenant_users(tenant_id, user_id) -- Foreign key reference to the tenant_users table
);
```
### **4. tasks**
The `tasks` table manages tasks assigned as part of the onboarding process. Tasks can be assigned to either customers or support members. It includes details about the task, its status, and timestamps for creation and completion. The embeddings on the task description helps AI to assist the customer on how to execute the task by looking at similar tasks on other projects.
```sql theme={null}
CREATE TABLE tasks (
tenant_id UUID, -- Identifier for the tenant (organization) to which the task belongs
task_id UUID, -- Unique identifier for the task
project_id UUID, -- Identifier for the project associated with the task
assigned_to UUID, -- Identifier for the person assigned the task (customer or support member)
description TEXT NOT NULL, -- Detailed description of the task
status VARCHAR(50) NOT NULL, -- Current status of the task (e.g., Not Started, In Progress, Completed)
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, -- Timestamp when the task was created
completed_at TIMESTAMP, -- Timestamp when the task was completed
vector_embedding VECTOR(768), -- Embedding vector for AI-related operations (e.g., task analysis)
PRIMARY KEY (tenant_id, task_id), -- Composite primary key combining tenant_id and task_id
FOREIGN KEY (tenant_id, project_id) REFERENCES projects(tenant_id, project_id), -- Foreign key reference to the projects table
FOREIGN KEY (tenant_id, assigned_to) REFERENCES tenant_users(tenant_id, user_id) -- Foreign key reference to the tenant_users table
);
```
### **5. knowledge\_base**
The `knowledge_base` table contains articles and resources that assist with the onboarding process. These articles can be accessed by both customers and support members and are used to provide guidance and answers during onboarding. The embeddings on the knowledge base helps to summarize data for both the support and the customer for onboarding.
```sql theme={null}
CREATE TABLE knowledge_base (
tenant_id UUID, -- Identifier for the tenant (organization) to which the knowledge base belongs
article_id UUID, -- Unique identifier for the article
title VARCHAR(100) NOT NULL, -- Title of the article
content TEXT NOT NULL, -- Content of the article
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, -- Timestamp when the article was created
vector_embedding VECTOR(768), -- Embedding vector for AI-related operations (e.g., content analysis)
PRIMARY KEY (tenant_id, article_id), -- Composite primary key combining tenant_id and article_id
FOREIGN KEY (tenant_id) REFERENCES tenants(tenant_id) -- Foreign key reference to the tenants table
);
```
### **6. notifications**
The `notifications` table handles notifications related to the onboarding process. It tracks messages sent to designated roles (like sales leaders) and includes details about the message and its read status.
```sql theme={null}
CREATE TABLE notifications (
tenant_id UUID, -- Identifier for the tenant (organization) to which the notification belongs
notification_id UUID, -- Unique identifier for the notification
recipient_id UUID, -- Identifier for the recipient of the notification (e.g., sales leader)
message TEXT NOT NULL, -- Content of the notification message
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, -- Timestamp when the notification was created
is_read BOOLEAN DEFAULT FALSE, -- Flag indicating whether the notification has been read
PRIMARY KEY (tenant_id, notification_id), -- Composite primary key combining tenant_id and notification_id
FOREIGN KEY (tenant_id, recipient_id) REFERENCES tenant_users(tenant_id, user_id) -- Foreign key reference to the tenant_users table
);
```
Each table schema is designed to adhere to the multitenant architecture, using `tenant_id` as a composite key and referencing other tables as needed.
### Full Script
```sql theme={null}
-- Customers Table
CREATE TABLE customers (
tenant_id UUID,
customer_id UUID,
name VARCHAR(100) NOT NULL,
contact_info JSONB,
PRIMARY KEY (tenant_id, customer_id),
FOREIGN KEY (tenant_id) REFERENCES tenants(id)
);
-- Projects Table
CREATE TABLE projects (
tenant_id UUID,
project_id UUID,
customer_id UUID,
name VARCHAR(100) NOT NULL,
description TEXT NOT NULL,
status VARCHAR(50) NOT NULL,
deadline TIMESTAMP,
progress NUMERIC(5, 2),
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
PRIMARY KEY (tenant_id, project_id),
FOREIGN KEY (tenant_id, customer_id) REFERENCES customers(tenant_id, customer_id)
);
-- Support Members Table
CREATE TABLE support_members (
tenant_id UUID,
support_member_id UUID,
PRIMARY KEY (tenant_id, support_member_id),
FOREIGN KEY (tenant_id, support_member_id) REFERENCES users.tenant_users(tenant_id, user_id)
);
-- Tasks Table
CREATE TABLE tasks (
tenant_id UUID,
task_id UUID,
project_id UUID,
assigned_to UUID, -- Can be either customer or support member
description TEXT NOT NULL,
status VARCHAR(50) NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
completed_at TIMESTAMP,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
PRIMARY KEY (tenant_id, task_id),
FOREIGN KEY (tenant_id, project_id) REFERENCES projects(tenant_id, project_id),
FOREIGN KEY (tenant_id, assigned_to) REFERENCES users.tenant_users(tenant_id, user_id)
);
-- Knowledge Base Table
CREATE TABLE knowledge_base (
tenant_id UUID,
article_id UUID,
title VARCHAR(100) NOT NULL,
content TEXT NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
PRIMARY KEY (tenant_id, article_id),
FOREIGN KEY (tenant_id) REFERENCES tenants(id)
);
-- Notifications Table
CREATE TABLE notifications (
tenant_id UUID,
notification_id UUID,
recipient_id UUID, -- Sales leader or any designated role
message TEXT NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
is_read BOOLEAN DEFAULT FALSE,
PRIMARY KEY (tenant_id, notification_id),
FOREIGN KEY (tenant_id, recipient_id) REFERENCES users.tenant_users(tenant_id, user_id)
);
```
# CalendIQ - Intelligent Calendar Management for Busy Lives
Source: https://thenile.dev/docs/getting-started/usecases/scheduler
The meeting scheduling application is designed to help organizations efficiently manage their employees' calendars, schedule meetings, and provide insights into how time is being spent. The application leverages AI to automate and optimize meeting scheduling, rescheduling, and time management.
### Key Features
1. **Calendar Management**
* Each employee has a personal calendar that tracks their availability and meetings.
* Calendars can be shared with others for collaboration and meeting scheduling.
2. **Meeting Scheduling**
* Users can schedule meetings by selecting available times from multiple employees' calendars.
* AI suggests optimal meeting times based on participants' availability.
3. **Meeting Tracking**
* The system tracks all scheduled meetings, including details such as participants, time, and location.
* Users receive reminders for upcoming meetings.
4. **Meeting Rescheduling**
* AI assists in rescheduling meetings to find the most optimal times, reducing scheduling conflicts.
5. **Statistics and Insights**
* Provides statistics on how employees' time is being spent in meetings.
* AI suggests ways to reduce time spent in meetings based on past data.
6. **AI-Powered Features**
* AI books meetings automatically based on employees' calendars.
* AI proposes ways to optimize meeting times and reduce time in meetings.
* AI helps reschedule meetings for optimal use of time.
### Postgres Schemas
### 1. calendars
Stores calendar entries for each employee, including availability and meetings.
```sql theme={null}
CREATE TABLE calendars (
tenant_id UUID,
calendar_id UUID DEFAULT gen_random_uuid(),
user_id UUID,
event_date TIMESTAMP NOT NULL,
event_type VARCHAR(50) NOT NULL, -- e.g., "meeting", "available", "unavailable"
event_details JSONB,
PRIMARY KEY (tenant_id, calendar_id),
FOREIGN KEY (tenant_id, user_id) REFERENCES users.tenant_users(tenant_id, user_id)
);
```
### 2. meetings
Stores meeting information, including participants, time, location, and status. The embeddings are stored for the details of each meetings. This helps the AI model to understand meeting distribution and how time is spent.
```sql theme={null}
CREATE TABLE meetings (
tenant_id UUID,
meeting_id UUID DEFAULT gen_random_uuid(),
organizer_id UUID,
meeting_time TIMESTAMP NOT NULL,
location VARCHAR(255),
status VARCHAR(50), -- e.g., "scheduled", "completed", "canceled"
details TEXT,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
PRIMARY KEY (tenant_id, meeting_id),
FOREIGN KEY (tenant_id, organizer_id) REFERENCES users.tenant_users(tenant_id, user_id)
);
```
### 3. meeting\_participants
Maps participants to meetings, tracking which employees are involved in each meeting.
```sql theme={null}
CREATE TABLE meeting_participants (
tenant_id UUID,
meeting_id UUID,
user_id UUID,
PRIMARY KEY (tenant_id, meeting_id, user_id),
FOREIGN KEY (tenant_id, meeting_id) REFERENCES meetings(tenant_id, meeting_id),
FOREIGN KEY (tenant_id, user_id) REFERENCES users.tenant_users(tenant_id, user_id)
);
```
### 4. reminders
Stores reminders for meetings, including the reminder time and message.
```sql theme={null}
CREATE TABLE reminders (
tenant_id UUID,
reminder_id UUID DEFAULT gen_random_uuid(),
meeting_id UUID,
reminder_time TIMESTAMP NOT NULL,
message TEXT,
PRIMARY KEY (tenant_id, reminder_id),
FOREIGN KEY (tenant_id, meeting_id) REFERENCES meetings(tenant_id, meeting_id)
);
```
### 5. meeting\_statistics
Stores statistics on meeting attendance and duration for analysis and optimization.
```sql theme={null}
CREATE TABLE meeting_statistics (
tenant_id UUID,
stat_id UUID DEFAULT gen_random_uuid(),
meeting_id UUID,
duration_minutes INT,
attended BOOLEAN,
PRIMARY KEY (tenant_id, stat_id),
FOREIGN KEY (tenant_id, meeting_id) REFERENCES meetings(tenant_id, meeting_id)
);
```
### 6. shared\_calendars
Stores information about calendars shared between users for collaborative scheduling.
```sql theme={null}
CREATE TABLE shared_calendars (
tenant_id UUID,
owner_id UUID,
shared_with_id UUID,
PRIMARY KEY (tenant_id, owner_id, shared_with_id),
FOREIGN KEY (tenant_id, owner_id) REFERENCES users.tenant_users(tenant_id, user_id),
FOREIGN KEY (tenant_id, shared_with_id) REFERENCES users.tenant_users(tenant_id, user_id)
);
```
This schema ensures that all relevant data for the meeting scheduling application is captured and organized efficiently, with AI capabilities integrated to provide advanced features and insights.
### Full Script
```sql theme={null}
-- Create Calendars Table
CREATE TABLE calendars (
tenant_id UUID,
calendar_id UUID DEFAULT gen_random_uuid(),
user_id UUID,
event_date TIMESTAMP NOT NULL,
event_type VARCHAR(50) NOT NULL, -- e.g., "meeting", "available", "unavailable"
event_details JSONB,
PRIMARY KEY (tenant_id, calendar_id),
FOREIGN KEY (tenant_id, user_id) REFERENCES users.tenant_users(tenant_id, user_id)
);
-- Create Meetings Table
CREATE TABLE meetings (
tenant_id UUID,
meeting_id UUID DEFAULT gen_random_uuid(),
organizer_id UUID,
meeting_time TIMESTAMP NOT NULL,
location VARCHAR(255),
status VARCHAR(50), -- e.g., "scheduled", "completed", "canceled"
details TEXT,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
PRIMARY KEY (tenant_id, meeting_id),
FOREIGN KEY (tenant_id, organizer_id) REFERENCES users.tenant_users(tenant_id, user_id)
);
-- Create Meeting Participants Table
CREATE TABLE meeting_participants (
tenant_id UUID,
meeting_id UUID,
user_id UUID,
PRIMARY KEY (tenant_id, meeting_id, user_id),
FOREIGN KEY (tenant_id, meeting_id) REFERENCES meetings(tenant_id, meeting_id),
FOREIGN KEY (tenant_id, user_id) REFERENCES users.tenant_users(tenant_id, user_id)
);
-- Create Reminders Table
CREATE TABLE reminders (
tenant_id UUID,
reminder_id UUID DEFAULT gen_random_uuid(),
meeting_id UUID,
reminder_time TIMESTAMP NOT NULL,
message TEXT,
PRIMARY KEY (tenant_id, reminder_id),
FOREIGN KEY (tenant_id, meeting_id) REFERENCES meetings(tenant_id, meeting_id)
);
-- Create Meeting Statistics Table
CREATE TABLE meeting_statistics (
tenant_id UUID,
stat_id UUID DEFAULT gen_random_uuid(),
meeting_id UUID,
duration_minutes INT,
attended BOOLEAN,
PRIMARY KEY (tenant_id, stat_id),
FOREIGN KEY (tenant_id, meeting_id) REFERENCES meetings(tenant_id, meeting_id)
);
-- Create Shared Calendars Table
CREATE TABLE shared_calendars (
tenant_id UUID,
owner_id UUID,
shared_with_id UUID,
PRIMARY KEY (tenant_id, owner_id, shared_with_id),
FOREIGN KEY (tenant_id, owner_id) REFERENCES users.tenant_users(tenant_id, user_id),
FOREIGN KEY (tenant_id, shared_with_id) REFERENCES users.tenant_users(tenant_id, user_id)
);
```
# SmartLearn: AI Intelligence for the Modern Classroom
Source: https://thenile.dev/docs/getting-started/usecases/schoology
This application aims to assist both students and teachers in managing courses, tracking academic progress, and enhancing the educational experience through AI-driven features. The application provides functionalities for tracking courses, grades, progress, feedback, and homework. For teachers, it enables the management of students, assignment of homework, feedback provision, and grade entry. AI features include student progress analysis, future performance improvement plans, feedback search, and automated test correction.
### Detailed Requirements
1. **Multitenancy:** Each school (tenant) has its own isolated data, ensuring data security and segregation.
2. **Course Management:** Students can enroll in courses, track their progress, and view grades.
3. **Grade Tracking:** Students' grades are recorded and can be analyzed over time.
4. **Progress Monitoring:** Students can track their academic progress.
5. **Feedback:** Teachers can provide feedback on students' performance.
6. **Homework Management:** Teachers can assign, track, and grade homework.
7. **AI Features:**
* Analyze student progress based on feedback and grades.
* Provide personalized improvement plans based on past performance.
* Assist teachers in searching through student feedback.
* Automatically correct handwritten tests using generative AI.
8. **User Management:** Students and teachers can belong to multiple tenants.
9. **Search Functionality:** Teachers can search for feedback and grades of all students.
### Postgres Schemas
### 1. courses
Stores information about each course. Both the teachers and students can ask questions about the course and teachers can generate new courses based on existing courses. For this purpose, generating embeddings for each course description would be useful. The application can then implement RAG to build use cases such as searching courses and generating new courses.
```sql theme={null}
CREATE TABLE courses (
tenant_id UUID,
course_id UUID DEFAULT gen_random_uuid(),
course_name VARCHAR(100) NOT NULL,
course_description TEXT,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed,
PRIMARY KEY (tenant_id, course_id),
FOREIGN KEY (tenant_id) REFERENCES tenants(id)
);
```
### 2. enrollments
Tracks which students are enrolled in which courses.
```sql theme={null}
CREATE TABLE enrollments (
tenant_id UUID,
enrollment_id UUID DEFAULT gen_random_uuid(),
course_id UUID,
student_id UUID,
enrollment_date TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, enrollment_id),
FOREIGN KEY (tenant_id, course_id) REFERENCES courses(tenant_id, course_id),
FOREIGN KEY (tenant_id, student_id) REFERENCES tenant_users(tenant_id, user_id)
);
```
### 3. grades
Stores grades for each student in each course.
```sql theme={null}
CREATE TABLE grades (
tenant_id UUID,
grade_id UUID DEFAULT gen_random_uuid(),
course_id UUID,
student_id UUID,
grade DECIMAL(5, 2),
grade_date TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, grade_id),
FOREIGN KEY (tenant_id, course_id) REFERENCES courses(tenant_id, course_id),
FOREIGN KEY (tenant_id, student_id) REFERENCES tenant_users(tenant_id, user_id)
);
```
### 4. feedback
Stores feedback from teachers for each student in each course. The feedbacks are extremely useful input to search for both students and teachers and learn about the progress. The chatbot can be used to ask questions about past feedback and even create plans for the future. So, having embeddings on the feedback helps with building RAG architecture.
```sql theme={null}
CREATE TABLE feedback (
tenant_id UUID,
feedback_id UUID DEFAULT gen_random_uuid(),
course_id UUID,
student_id UUID,
teacher_id UUID,
feedback_text TEXT,
feedback_date TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
PRIMARY KEY (tenant_id, feedback_id),
FOREIGN KEY (tenant_id, course_id) REFERENCES courses(tenant_id, course_id),
FOREIGN KEY (tenant_id, student_id) REFERENCES tenant_users(tenant_id, user_id),
FOREIGN KEY (tenant_id, teacher_id) REFERENCES tenant_users(tenant_id, user_id)
);
```
### 5. homework
Stores homework assignments for each course. The teachers can have AI generate solutions for the homework exercises. They can also leverage AI to help correct solutions based on the input questions. The embeddings here are on the homework questions and not the solutions themselves. This is useful for AI to generate solutions and even generate similar homeworks in the future.
```sql theme={null}
CREATE TABLE homework (
tenant_id UUID,
homework_id UUID DEFAULT gen_random_uuid(),
course_id UUID,
teacher_id UUID,
homework_description TEXT,
due_date DATE,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
PRIMARY KEY (tenant_id, homework_id),
FOREIGN KEY (tenant_id, course_id) REFERENCES courses(tenant_id, course_id),
FOREIGN KEY (tenant_id, teacher_id) REFERENCES tenant_users(tenant_id, user_id)
);
```
### 6. homework\_submissions
Tracks homework submissions by students. As mentioned above, these submissions are a file and the embeddings for the file are stored in the table. The table only tracks the file location and the actual file can be in some object store.
```sql theme={null}
CREATE TABLE homework_submissions (
tenant_id UUID,
submission_id UUID DEFAULT gen_random_uuid(),
homework_id UUID,
student_id UUID,
submission_date TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
submission_file BYTEA,
grade DECIMAL(5, 2),
feedback TEXT,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
PRIMARY KEY (tenant_id, submission_id),
FOREIGN KEY (tenant_id, homework_id) REFERENCES homework(tenant_id, homework_id),
FOREIGN KEY (tenant_id, student_id) REFERENCES tenant_users(tenant_id, user_id)
);
```
### Complete SQL Script
```sql theme={null}
-- Create courses table
CREATE TABLE courses (
tenant_id UUID,
course_id UUID DEFAULT gen_random_uuid(),
course_name VARCHAR(100) NOT NULL,
course_description TEXT,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed,
PRIMARY KEY (tenant_id, course_id),
FOREIGN KEY (tenant_id) REFERENCES tenants(id)
);
-- Create enrollments table
CREATE TABLE enrollments (
tenant_id UUID,
enrollment_id UUID DEFAULT gen_random_uuid(),
course_id UUID,
student_id UUID,
enrollment_date TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, enrollment_id),
FOREIGN KEY (tenant_id, course_id) REFERENCES courses(tenant_id, course_id),
FOREIGN KEY (tenant_id, student_id) REFERENCES tenant_users(tenant_id, user_id)
);
-- Create grades table
CREATE TABLE grades (
tenant_id UUID,
grade_id UUID DEFAULT gen_random_uuid(),
course_id UUID,
student_id UUID,
grade DECIMAL(5, 2),
grade_date TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, grade_id),
FOREIGN KEY (tenant_id, course_id) REFERENCES courses(tenant_id, course_id),
FOREIGN KEY (tenant_id, student_id) REFERENCES tenant_users(tenant_id, user_id)
);
-- Create feedback table
CREATE TABLE feedback (
tenant_id UUID,
feedback_id UUID DEFAULT gen_random_uuid(),
course_id UUID,
student_id UUID,
teacher_id UUID,
feedback_text TEXT,
feedback_date TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
PRIMARY KEY (tenant_id, feedback_id),
FOREIGN KEY (tenant_id, course_id) REFERENCES courses(tenant_id, course_id),
FOREIGN KEY (tenant_id, student_id) REFERENCES tenant_users(tenant_id, user_id),
FOREIGN KEY (tenant_id, teacher_id) REFERENCES tenant_users(tenant_id, user_id)
);
-- Create homework table
CREATE TABLE homework (
tenant_id UUID,
homework_id UUID DEFAULT gen_random_uuid(),
course_id UUID,
teacher_id UUID,
homework_description TEXT,
due_date DATE,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
PRIMARY KEY (tenant_id, homework_id),
FOREIGN KEY (tenant_id, course_id) REFERENCES courses(tenant_id, course_id),
FOREIGN KEY (tenant_id, teacher_id) REFERENCES tenant_users(tenant_id, user_id)
);
-- Create homework_submissions table
CREATE TABLE homework_submissions (
tenant_id UUID,
submission_id UUID DEFAULT gen_random_uuid(),
homework_id UUID,
student_id UUID,
submission_date TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
submission_file BYTEA,
grade DECIMAL(5, 2),
feedback TEXT,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
PRIMARY KEY (tenant_id, submission_id),
FOREIGN KEY (tenant_id, homework_id) REFERENCES homework(tenant_id, homework_id),
FOREIGN KEY (tenant_id, student_id) REFERENCES tenant_users(tenant_id, user_id)
);
```
This script sets up the necessary tables and relationships to support a robust and scalable multitenant school course management application with advanced AI capabilities, including course management, grade tracking, feedback, homework management, and AI-driven features for progress analysis and automated test correction.
# Slack++ - Built with AI to power work
Source: https://thenile.dev/docs/getting-started/usecases/slack
Slack++ is a multitenant application designed to provide a robust messaging platform similar to Slack but with additional AI-powered features. The primary tenants of this system are organizations, each identified by a `tenant_id`. The key features include:
1. **Channels:** Users can create and join channels to facilitate team communication.
2. **Messages:** Users can send and receive messages within channels.
3. **Direct Messages:** Users can communicate privately with one another.
4. **AI Summarization:** The AI system can summarize conversations within channels.
5. **Message Search:** Users can search for messages within a channel and across channels.
6. **Theme Highlighting:** The AI chatbot can highlight themes from discussions in a channel.
7. **User Management:** Manage users and their membership in channels.
Each table will include `tenant_id` as part of its primary key, and all IDs will be of type UUID.
### Postgres Schemas
Note that the tenants, users and tenant\_users tables references in these schemas are already built-in within Nile's
Postgres database.
### 1. channels
Stores channel information for each tenant. Users can be part of multiple channels and conversation happens within channels.
```sql theme={null}
CREATE TABLE channels (
tenant_id UUID,
channel_id UUID DEFAULT gen_random_uuid(),
channel_name VARCHAR(100) NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, channel_id),
FOREIGN KEY (tenant_id) REFERENCES tenants(id)
);
```
### 2. channel\_users
Maps users to channels within a tenant. The users within a tenant can be part of multiple channels.
```sql theme={null}
CREATE TABLE channel_users (
tenant_id UUID,
channel_id UUID,
user_id UUID,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, channel_id, user_id),
FOREIGN KEY (tenant_id, channel_id) REFERENCES channels(tenant_id, channel_id),
FOREIGN KEY (tenant_id, user_id) REFERENCES tenant_users(tenant_id, user_id)
);
```
### 3. messages
Stores messages within channels along with vector embeddings for AI functionalities. Note that this example calculates embeddings for each message. In practice, you would not do this since it would blow up the number of embeddings and lack full context. A better option is to chunk the messages around a time range into one embedding. You could optionally have a messagechunk table that track each chunk and stores embeddings for them.
```sql theme={null}
CREATE TABLE messages (
tenant_id UUID,
message_id UUID DEFAULT gen_random_uuid(),
channel_id UUID,
user_id UUID,
content TEXT NOT NULL,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, message_id),
FOREIGN KEY (tenant_id, channel_id) REFERENCES channels(tenant_id, channel_id),
FOREIGN KEY (tenant_id, user_id) REFERENCES tenant_users(tenant_id, user_id)
);
```
### 4. direct\_messages
Stores private messages between users along with vector embeddings for AI functionalities. Like the messages table, you would want to use a chunking approach for messages.
```sql theme={null}
CREATE TABLE direct_messages (
tenant_id UUID,
dm_id UUID DEFAULT gen_random_uuid(),
sender_id UUID,
receiver_id UUID,
content TEXT NOT NULL,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, dm_id),
FOREIGN KEY (tenant_id, sender_id) REFERENCES tenant_users(tenant_id, user_id),
FOREIGN KEY (tenant_id, receiver_id) REFERENCES tenant_users(tenant_id, user_id)
);
```
These schemas provide the foundation for a multitenant Slack++ application with AI features, ensuring each table follows the multitenant structure with `tenant_id` and uses UUIDs for primary keys. The `vector_embedding` column is included in the `messages` and `direct_messages` tables to support AI functionalities like summarization and search.
### Full Script
```sql theme={null}
-- Create channels table
CREATE TABLE channels (
tenant_id UUID,
channel_id UUID DEFAULT gen_random_uuid(),
channel_name VARCHAR(100) NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, channel_id),
FOREIGN KEY (tenant_id) REFERENCES tenants(id)
);
-- Create channel_users table to map users to channels within a tenant
CREATE TABLE channel_users (
tenant_id UUID,
channel_id UUID,
user_id UUID,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, channel_id, user_id),
FOREIGN KEY (tenant_id, channel_id) REFERENCES channels(tenant_id, channel_id),
FOREIGN KEY (tenant_id, user_id) REFERENCES tenant_users(tenant_id, user_id)
);
-- Create messages table with vector embedding for AI functionalities
CREATE TABLE messages (
tenant_id UUID,
message_id UUID DEFAULT gen_random_uuid(),
channel_id UUID,
user_id UUID,
content TEXT NOT NULL,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, message_id),
FOREIGN KEY (tenant_id, channel_id, user_id) REFERENCES channel_users(tenant_id, channel_id, user_id)
);
-- Create direct_messages table with vector embedding for AI functionalities
CREATE TABLE direct_messages (
tenant_id UUID,
dm_id UUID DEFAULT gen_random_uuid(),
sender_id UUID,
receiver_id UUID,
content TEXT NOT NULL,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, dm_id),
FOREIGN KEY (tenant_id, sender_id) REFERENCES tenant_users(tenant_id, user_id),
FOREIGN KEY (tenant_id, receiver_id) REFERENCES tenant_users(tenant_id, user_id)
);
```
# SupplyAI - AI-Powered Efficiency for Your Supply Chain
Source: https://thenile.dev/docs/getting-started/usecases/supply
The Supply Chain Management (SCM) application is designed to streamline the procurement, shipping, and sales processes for businesses. It provides end-to-end visibility and control over the supply chain, from placing orders for supplies to choosing shipment options and selling products through platforms like Amazon or Shopify. The application incorporates AI features to optimize costs, manage unexpected delays, and suggest new vendors and strategies.
### Key Features
1. **Order Management:**
* Place orders for supplies from global vendors.
* Track order status and history.
* Integrate with vendor systems for real-time updates.
2. **Shipping Management:**
* Choose from various shipment options.
* Track shipment progress.
* Integrate with fulfillment services (e.g., FedEx, UPS).
3. **Sales Integration:**
* Choose sales channels like Amazon or Shopify.
* Track sales orders and inventory.
4. **AI Features:**
* Optimize shipping costs based on past trends.
* Answer questions about current shipments and unexpected delays.
* Propose new vendors and strategies to reduce costs.
5. **Reporting and Analytics:**
* Generate reports on order status, shipping performance, and sales.
* Provide insights on cost-saving opportunities.
### Postgres Schemas
### 1. vendors
This table stores information about vendors from whom supplies are purchased. Each vendor has a unique identifier, a reference to the tenant, and vendor details including contact information.
```sql theme={null}
CREATE TABLE supply.vendors (
tenant_id UUID,
vendor_id UUID,
name VARCHAR(100) NOT NULL,
contact_info JSONB,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
PRIMARY KEY (tenant_id, vendor_id),
FOREIGN KEY (tenant_id) REFERENCES tenants(id)
);
```
### 2. orders
This table stores information about orders placed to buy supplies. Each order has a unique identifier, a reference to the tenant and vendor, order details, and a status.
```sql theme={null}
CREATE TABLE supply.orders (
tenant_id UUID,
order_id UUID,
vendor_id UUID,
order_date TIMESTAMP NOT NULL,
status VARCHAR(50) NOT NULL,
total_amount NUMERIC(15, 2) NOT NULL,
PRIMARY KEY (tenant_id, order_id),
FOREIGN KEY (tenant_id, vendor_id) REFERENCES supply.vendors(tenant_id, vendor_id)
);
```
### 3. order\_items
This table tracks individual items within an order. Each order item has a unique identifier, a reference to the tenant and order, and includes product details, quantity, and price.
```sql theme={null}
CREATE TABLE supply.order_items (
tenant_id UUID,
order_item_id UUID,
order_id UUID,
product_name VARCHAR(100) NOT NULL,
quantity INT NOT NULL,
price NUMERIC(15, 2) NOT NULL,
PRIMARY KEY (tenant_id, order_item_id),
FOREIGN KEY (tenant_id, order_id) REFERENCES supply.orders(tenant_id, order_id)
);
```
### 4. shipments
This table tracks the shipment details of orders, including the shipment status, carrier, and estimated delivery date. The embeddings are calculated on the tracking information. This helps AI to suggest possible delays and also see trends across shipments and propose best routes in the future.
```sql theme={null}
CREATE TABLE supply.shipments (
tenant_id UUID,
shipment_id UUID,
order_id UUID,
shipment_date TIMESTAMP,
delivery_date TIMESTAMP,
status VARCHAR(50),
tracking_info JSONB,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
PRIMARY KEY (tenant_id, shipment_id),
FOREIGN KEY (tenant_id, order_id) REFERENCES supply.orders(tenant_id, order_id)
);
```
### 5. fulfillment\_services
This table stores information about the fulfillment services integrated with the application. Each service has a unique identifier and details about the service. The embeddings are calculated on the fulfillment information. This can be used by AI to identify similar services with cheaper and more efficient operation.
```sql theme={null}
CREATE TABLE supply.fulfillment_services (
tenant_id UUID,
service_id UUID,
name VARCHAR(100) NOT NULL,
contact_info JSONB,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, service_id),
FOREIGN KEY (tenant_id) REFERENCES tenants(id)
);
```
### 6. sales\_channels
This table stores information about the sales channels used to sell products, such as Amazon or Shopify. Each channel has a unique identifier and details about the channel.
```sql theme={null}
CREATE TABLE supply.sales_channels (
tenant_id UUID,
channel_id UUID,
name VARCHAR(100) NOT NULL,
contact_info JSONB,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, channel_id),
FOREIGN KEY (tenant_id) REFERENCES tenants(id)
);
```
### 7. sales\_orders
This table tracks sales orders, including details such as the order date, status, total amount, and the sales channel used.
```sql theme={null}
CREATE TABLE supply.sales_orders (
tenant_id UUID,
sales_order_id UUID,
channel_id UUID,
order_date TIMESTAMP NOT NULL,
status VARCHAR(50) NOT NULL,
total_amount NUMERIC(15, 2) NOT NULL,
PRIMARY KEY (tenant_id, sales_order_id),
FOREIGN KEY (tenant_id, channel_id) REFERENCES supply.sales_channels(tenant_id, channel_id)
);
```
### 8. sales\_order\_items
This table tracks individual items within a sales order, including product details, quantity, and price.
```sql theme={null}
CREATE TABLE supply.sales_order_items (
tenant_id UUID,
sales_order_item_id UUID,
sales_order_id UUID,
product_name VARCHAR(100) NOT NULL,
quantity INT NOT NULL,
price NUMERIC(15, 2) NOT NULL,
PRIMARY KEY (tenant_id, sales_order_item_id),
FOREIGN KEY (tenant_id, sales_order_id) REFERENCES supply.sales_orders(tenant_id, sales_order_id)
);
```
These tables collectively enable the application to manage various aspects of the supply chain, from ordering and shipping to sales and payments, while leveraging AI to provide insights and recommendations.
### Full Script
```sql theme={null}
-- Vendors Table
CREATE TABLE supply.vendors (
tenant_id UUID,
vendor_id UUID,
name VARCHAR(100) NOT NULL,
contact_info JSONB,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
PRIMARY KEY (tenant_id, vendor_id),
FOREIGN KEY (tenant_id) REFERENCES tenants(id)
);
-- Orders Table
CREATE TABLE supply.orders (
tenant_id UUID,
order_id UUID,
vendor_id UUID,
order_date TIMESTAMP NOT NULL,
status VARCHAR(50) NOT NULL,
total_amount NUMERIC(15, 2) NOT NULL,
PRIMARY KEY (tenant_id, order_id),
FOREIGN KEY (tenant_id, vendor_id) REFERENCES supply.vendors(tenant_id, vendor_id)
);
-- Order Items Table
CREATE TABLE supply.order_items (
tenant_id UUID,
order_item_id UUID,
order_id UUID,
product_name VARCHAR(100) NOT NULL,
quantity INT NOT NULL,
price NUMERIC(15, 2) NOT NULL,
PRIMARY KEY (tenant_id, order_item_id),
FOREIGN KEY (tenant_id, order_id) REFERENCES supply.orders(tenant_id, order_id)
);
-- Shipments Table
CREATE TABLE supply.shipments (
tenant_id UUID,
shipment_id UUID,
order_id UUID,
shipment_date TIMESTAMP,
delivery_date TIMESTAMP,
status VARCHAR(50),
tracking_info JSONB,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
PRIMARY KEY (tenant_id, shipment_id),
FOREIGN KEY (tenant_id, order_id) REFERENCES supply.orders(tenant_id, order_id)
);
-- Fulfillment Services Table
CREATE TABLE supply.fulfillment_services (
tenant_id UUID,
service_id UUID,
name VARCHAR(100) NOT NULL,
contact_info JSONB,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, service_id),
FOREIGN KEY (tenant_id) REFERENCES tenants(id)
);
-- Sales Channels Table
CREATE TABLE supply.sales_channels (
tenant_id UUID,
channel_id UUID,
name VARCHAR(100) NOT NULL,
contact_info JSONB,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, channel_id),
FOREIGN KEY (tenant_id) REFERENCES tenants(id)
);
-- Sales Orders Table
CREATE TABLE supply.sales_orders (
tenant_id UUID,
sales_order_id UUID,
channel_id UUID,
order_date TIMESTAMP NOT NULL,
status VARCHAR(50) NOT NULL,
total_amount NUMERIC(15, 2) NOT NULL,
PRIMARY KEY (tenant_id, sales_order_id),
FOREIGN KEY (tenant_id, channel_id) REFERENCES supply.sales_channels(tenant_id, channel_id)
);
-- Sales Order Items Table
CREATE TABLE supply.sales_order_items (
tenant_id UUID,
sales_order_item_id UUID,
sales_order_id UUID,
product_name VARCHAR(100) NOT NULL,
quantity INT NOT NULL,
price NUMERIC(15, 2) NOT NULL,
PRIMARY KEY (tenant_id, sales_order_item_id),
FOREIGN KEY (tenant_id, sales_order_id) REFERENCES supply.sales_orders(tenant_id, sales_order_id)
);
```
# CustomerDesk AI - AI-Powered Efficiency for Your Supply Chain
Source: https://thenile.dev/docs/getting-started/usecases/support
### 1. Multitenancy
* **Data Isolation:** Each tenant's data is stored separately, ensuring that one organization's data is not accessible to another.
* **Scalability:** The system can handle multiple tenants efficiently, scaling as the number of organizations increases.
* **Security:** Enhanced security measures are in place to protect tenant data and ensure compliance with data protection regulations.
### 2. User Roles
* **Support Members:**
* **Reply to Support Tickets:** Support members can view and respond to support tickets raised by customers.
* **Summarize Tickets:** Using AI, support members can generate concise summaries of lengthy support tickets to quickly understand the issue.
* **AI-assisted Reply Drafting:** AI helps draft appropriate responses to customer queries, making the process more efficient.
* **Prioritization:** Support members can set priorities for tickets (e.g., high, medium, low) to manage workflow better.
* **Track Status:** The status of each ticket (open, in-progress, resolved, closed) can be tracked to monitor progress.
* **Create Knowledge Base Articles:** Support members can create articles to help customers self-service their issues.
* **AI-assisted Article Drafting:** AI aids in drafting articles, ensuring they are comprehensive and easy to understand.
* **Customers:**
* **Create Support Tickets:** Customers can raise support tickets to get help with their issues.
* **Respond to Support Questions:** Customers can interact with support members by replying to their questions within the ticket.
* **Browse Knowledge Base Articles:** Customers have access to a knowledge base where they can find articles to solve common problems.
* **Ask Questions on Articles:** Customers can ask questions on knowledge base articles for further clarification.
* **Mark Ticket Status:** Customers can update the status of their tickets (e.g., resolved if the issue is fixed).
### 3. Support Tickets
* **Creation:** Customers can create support tickets through the customer portal.
* **Updating:** Both support members and customers can update the tickets with new information.
* **Tracking:** Every ticket is tracked from creation to resolution, including the history of all interactions.
* **AI-assisted Reply Drafting and Summarization:** AI helps support members draft replies and summarize tickets to improve response times and clarity.
* **Prioritization and Status Tracking:** Tickets can be prioritized, and their status can be tracked to ensure timely resolution.
### 4. Knowledge Base
* **Article Creation and Updating:** Support members can create and update knowledge base articles to provide self-help resources to customers.
* **AI-assisted Drafting:** AI assists in drafting articles to ensure they are well-written and informative.
* **Browsing and Interaction:** Customers can browse articles and ask questions on them, facilitating a better understanding of the solutions provided.
### Postgres Schemas
### 1. support\_members
Tracks support members who are part of the support team. The support member belongs to a specific tenant.
```sql theme={null}
CREATE TABLE support_members (
tenant_id UUID,
member_id UUID,
role VARCHAR(50) NOT NULL,
PRIMARY KEY (tenant_id, member_id),
FOREIGN KEY (tenant_id, member_id) REFERENCES users.tenant_users(tenant_id, user_id)
);
```
### 2. customers
Track a list of customers that a particular tenant is managing through support. This tracks all the details about the customer that a particular tenant is managing.
```sql theme={null}
CREATE TABLE customers (
tenant_id UUID,
customer_id UUID DEFAULT gen_random_uuid(),
name VARCHAR(100) NOT NULL,
contact_email VARCHAR(100) NOT NULL,
contact_phone VARCHAR(20),
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, customer_id),
FOREIGN KEY (tenant_id) REFERENCES tenants(id)
);
```
### 3. customer\_employees
Tracks customer employees who are part of the customer team. The customer employees are part of a particular customer. It tracks all the information about the employee.
```sql theme={null}
CREATE TABLE customer_employees (
tenant_id UUID,
employee_id UUID,
customer_id UUID,
department VARCHAR(100),
PRIMARY KEY (tenant_id, employee_id),
FOREIGN KEY (tenant_id, customer_id) REFERENCES customers(tenant_id, customer_id),
FOREIGN KEY (tenant_id, employee_id) REFERENCES tenant_users(tenant_id, user_id)
);
```
### 4. support\_tickets
Tracks support tickets created by customers. The table tracks all the information about a particular ticket. To implement summarization and search on tickets, the embeddings are calculated on the descriptions of the ticket. In an actual application, you may want to chunk the description to create embeddings.
```sql theme={null}
CREATE TABLE support_tickets (
tenant_id UUID,
ticket_id UUID DEFAULT gen_random_uuid(),
customer_id UUID,
title VARCHAR(255) NOT NULL,
description TEXT NOT NULL,
status VARCHAR(50) DEFAULT 'open',
priority VARCHAR(50),
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
PRIMARY KEY (tenant_id, ticket_id),
FOREIGN KEY (tenant_id, customer_id) REFERENCES customers(tenant_id, customer_id)
);
```
### 5. ticket\_comments
Tracks comments on support tickets. The table tracks all the comments made by both the support member and the customer employee. This is why user\_id is used to reference the owner of the comment. The comments can also be auto filled by AI and to do this, we calculate the embeddings of all the past comments to feed into the AI model.
```sql theme={null}
CREATE TABLE ticket_comments (
tenant_id UUID,
comment_id UUID DEFAULT gen_random_uuid(),
ticket_id UUID,
user_id UUID,
comment_text TEXT NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
PRIMARY KEY (tenant_id, comment_id),
FOREIGN KEY (tenant_id, ticket_id) REFERENCES support_tickets(tenant_id, ticket_id),
FOREIGN KEY (tenant_id, user_id) REFERENCES tenant_users(tenant_id, user_id)
);
```
### 6. knowledge\_base
Tracks knowledge base articles created by the support team. This table is shared across customers within a tenant. These knowledge bases have embeddings calculated for each article. In a real world application, the knowledge based are chunked to create multiple embeddings per article.
```sql theme={null}
CREATE TABLE knowledge_base (
article_id UUID DEFAULT gen_random_uuid(),
tenant_id UUID,
support_member_id UUID,
title VARCHAR(255) NOT NULL,
content TEXT NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
PRIMARY KEY (tenant_id, article_id),
FOREIGN KEY (tenant_id, support_member_id) REFERENCES support_members(tenant_id, member_id)
);
```
### Complete SQL Script
```sql theme={null}
-- Create support_members table
CREATE TABLE support_members (
tenant_id UUID,
member_id UUID,
role VARCHAR(50) NOT NULL,
PRIMARY KEY (tenant_id, member_id),
FOREIGN KEY (tenant_id, member_id) REFERENCES users.tenant_users(tenant_id, user_id)
);
-- Create customers table
CREATE TABLE customers (
tenant_id UUID,
customer_id UUID DEFAULT gen_random_uuid(),
name VARCHAR(100) NOT NULL,
contact_email VARCHAR(100) NOT NULL,
contact_phone VARCHAR(20),
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, customer_id),
FOREIGN KEY (tenant_id) REFERENCES tenants(id)
);
-- Create customer_employees table
CREATE TABLE customer_employees (
tenant_id UUID,
employee_id UUID,
customer_id UUID,
department VARCHAR(100),
PRIMARY KEY (tenant_id, employee_id),
FOREIGN KEY (tenant_id, customer_id) REFERENCES customers(tenant_id, customer_id),
FOREIGN KEY (tenant_id, employee_id) REFERENCES tenant_users(tenant_id, user_id)
);
-- Create support_tickets table
CREATE TABLE support_tickets (
tenant_id UUID,
ticket_id UUID DEFAULT gen_random_uuid(),
customer_id UUID,
title VARCHAR(255) NOT NULL,
description TEXT NOT NULL,
status VARCHAR(50) DEFAULT 'open',
priority VARCHAR(50),
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
PRIMARY KEY (tenant_id, ticket_id),
FOREIGN KEY (tenant_id, customer_id) REFERENCES customers(tenant_id, customer_id)
);
-- Create ticket_comments table
CREATE TABLE ticket_comments (
tenant_id UUID,
comment_id UUID DEFAULT gen_random_uuid(),
ticket_id UUID,
user_id UUID,
comment_text TEXT NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
PRIMARY KEY (tenant_id, comment_id),
FOREIGN KEY (tenant_id, ticket_id) REFERENCES support_tickets(tenant_id, ticket_id),
FOREIGN KEY (tenant_id, user_id) REFERENCES tenant_users(tenant_id, user_id)
);
-- Create knowledge_base table
CREATE TABLE knowledge_base (
article_id UUID DEFAULT gen_random_uuid(),
tenant_id UUID,
support_member_id UUID,
title VARCHAR(255) NOT NULL,
content TEXT NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
vector_embedding VECTOR(768), -- Adjust the dimensions as needed
PRIMARY KEY (tenant_id, article_id),
FOREIGN KEY (tenant_id, support_member_id) REFERENCES support_members(tenant_id, member_id)
);
```
# AITravelMate - your business trips managed by AI
Source: https://thenile.dev/docs/getting-started/usecases/tripactions
The AI-native travel planning SaaS will leverage Generative AI to provide personalized and efficient travel itineraries. The system must support multitenancy, allowing multiple tenants to use the application independently with their own data, preferences, and policies. Each tenant's data should be isolated and securely managed. The system needs to track a list of draft itineraries that are not yet booked, a list of completed bookings, the ability to remind users when their travel date is approaching, and store travel preferences for each tenant to inform itinerary decisions. Additionally, the system should provide analytics on past travel spends. Security and privacy are paramount; hence, the system must comply with data protection regulations and implement robust authentication and authorization mechanisms. Finally, the architecture should be scalable to handle high traffic volumes and flexible to integrate with third-party services for real-time updates on travel information.
### PostgreSQL Schemas
### 1. tenants\_preferences
This table stores various preferences for each tenant, which the AI uses to generate personalized travel plans. Preferences are stored as key-value pairs in JSON format.
```sql theme={null}
CREATE TABLE tenant_preferences (
tenant_id UUID,
preference_id UUID DEFAULT gen_random_uuid(),
preference_key VARCHAR(100) NOT NULL,
preference_value JSONB NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, preference_id),
FOREIGN KEY (tenant_id) REFERENCES tenants(id)
);
```
### 2. user\_travel\_history
This table tracks the travel history of users per tenant. It includes details such as travel dates, destinations, activities, and total spend. This information is useful for generating analytics and improving future travel plans.
```sql theme={null}
CREATE TABLE user_travel_history (
tenant_id UUID,
history_id UUID DEFAULT gen_random_uuid(),
user_id UUID,
travel_date DATE NOT NULL,
destination VARCHAR(100) NOT NULL,
activities JSONB,
total_spend DECIMAL(10, 2) NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, history_id),
FOREIGN KEY (tenant_id,user_id) REFERENCES users.tenant_users(tenant_id,user_id)
);
```
### 3. itineraries
Itinerary Schema:\*\* This table stores draft and confirmed itineraries for users per tenant. It includes details like start and end dates, destinations, and a flag indicating whether the itinerary is a draft.
```sql theme={null}
CREATE TABLE itineraries (
tenant_id UUID,
itinerary_id UUID DEFAULT gen_random_uuid(),
user_id UUID,
start_date DATE NOT NULL,
end_date DATE NOT NULL,
destinations JSONB,
is_draft BOOLEAN DEFAULT TRUE,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, itinerary_id),
FOREIGN KEY (tenant_id,user_id) REFERENCES users.tenant_users(tenant_id,user_id)
);
```
### 4. itinerary\_details
This table provides detailed daily activities for each itinerary. It includes the date, type of activity, and specific details about the activity, stored in JSON format.
```sql theme={null}
CREATE TABLE itinerary_details (
tenant_id UUID,
detail_id UUID DEFAULT gen_random_uuid(),
itinerary_id UUID,
date DATE NOT NULL,
activity_type VARCHAR(50),
activity_details JSONB,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, detail_id),
FOREIGN KEY (tenant_id, itinerary_id) REFERENCES itineraries(tenant_id,itinerary_id)
);
```
### 5. bookings
This table tracks travel bookings made by users per tenant. It includes details about the booking type (e.g., flight, hotel), booking details, and status.
```sql theme={null}
CREATE TABLE bookings (
tenant_id UUID,
booking_id UUID DEFAULT gen_random_uuid(),
user_id UUID,
itinerary_id UUID,
booking_type VARCHAR(50) NOT NULL,
booking_details JSONB NOT NULL,
status VARCHAR(20) DEFAULT 'confirmed',
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, booking_id),
FOREIGN KEY (tenant_id,user_id) REFERENCES users.tenant_users(tenant_id,user_id),
FOREIGN KEY (tenant_id, itinerary_id) REFERENCES itineraries(tenant_id,itinerary_id)
);
```
### 6. booking\_reminders
This table stores reminders for users about their upcoming bookings. It includes the reminder date, message, and a flag indicating whether the reminder has been sent.
```sql theme={null}
CREATE TABLE booking_reminders (
tenant_id UUID,
reminder_id UUID DEFAULT gen_random_uuid(),
booking_id UUID,
user_id UUID,
reminder_date DATE NOT NULL,
message TEXT NOT NULL,
sent BOOLEAN DEFAULT FALSE,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, reminder_id),
FOREIGN KEY (tenant_id,user_id) REFERENCES users.tenant_users(tenant_id,user_id),
FOREIGN KEY (tenant_id,booking_id) REFERENCES bookings(tenant_id,booking_id)
);
```
### 7. travel\_analytics
This table stores analytics data about past travel spends for users per tenant. It includes the travel date, total spend, and a breakdown of spends by categories stored in JSON format.
```sql theme={null}
CREATE TABLE travel_analytics (
tenant_id UUID,
analytics_id UUID DEFAULT gen_random_uuid(),
user_id UUID,
travel_date DATE NOT NULL,
total_spend DECIMAL(10, 2) NOT NULL,
categories_spend JSONB NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, analytics_id),
FOREIGN KEY (tenant_id,user_id) REFERENCES users.tenant_users(tenant_id,user_id)
);
```
### Full Script
```sql theme={null}
CREATE TABLE tenant_preferences (
tenant_id UUID,
preference_id UUID DEFAULT gen_random_uuid(),
preference_key VARCHAR(100) NOT NULL,
preference_value JSONB NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, preference_id),
FOREIGN KEY (tenant_id) REFERENCES tenants(id)
);
CREATE TABLE user_travel_history (
tenant_id UUID,
history_id UUID DEFAULT gen_random_uuid(),
user_id UUID,
travel_date DATE NOT NULL,
destination VARCHAR(100) NOT NULL,
activities JSONB,
total_spend DECIMAL(10, 2) NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, history_id),
FOREIGN KEY (tenant_id,user_id) REFERENCES users.tenant_users(tenant_id,user_id)
);
CREATE TABLE itineraries (
tenant_id UUID,
itinerary_id UUID DEFAULT gen_random_uuid(),
user_id UUID,
start_date DATE NOT NULL,
end_date DATE NOT NULL,
destinations JSONB,
is_draft BOOLEAN DEFAULT TRUE,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, itinerary_id),
FOREIGN KEY (tenant_id,user_id) REFERENCES users.tenant_users(tenant_id,user_id)
);
CREATE TABLE itinerary_details (
tenant_id UUID,
detail_id UUID DEFAULT gen_random_uuid(),
itinerary_id UUID,
date DATE NOT NULL,
activity_type VARCHAR(50),
activity_details JSONB,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, detail_id),
FOREIGN KEY (tenant_id, itinerary_id) REFERENCES itineraries(tenant_id,itinerary_id)
);
CREATE TABLE bookings (
tenant_id UUID,
booking_id UUID DEFAULT gen_random_uuid(),
user_id UUID,
itinerary_id UUID,
booking_type VARCHAR(50) NOT NULL,
booking_details JSONB NOT NULL,
status VARCHAR(20) DEFAULT 'confirmed',
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, booking_id),
FOREIGN KEY (tenant_id,user_id) REFERENCES users.tenant_users(tenant_id,user_id),
FOREIGN KEY (tenant_id, itinerary_id) REFERENCES itineraries(tenant_id,itinerary_id)
);
CREATE TABLE booking_reminders (
tenant_id UUID,
reminder_id UUID DEFAULT gen_random_uuid(),
booking_id UUID,
user_id UUID,
reminder_date DATE NOT NULL,
message TEXT NOT NULL,
sent BOOLEAN DEFAULT FALSE,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, reminder_id),
FOREIGN KEY (tenant_id,user_id) REFERENCES users.tenant_users(tenant_id,user_id),
FOREIGN KEY (tenant_id,booking_id) REFERENCES bookings(tenant_id,booking_id)
);
CREATE TABLE travel_analytics (
tenant_id UUID,
analytics_id UUID DEFAULT gen_random_uuid(),
user_id UUID,
travel_date DATE NOT NULL,
total_spend DECIMAL(10, 2) NOT NULL,
categories_spend JSONB NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (tenant_id, analytics_id),
FOREIGN KEY (tenant_id,user_id) REFERENCES users.tenant_users(tenant_id,user_id)
);
```
# What is Nile
Source: https://thenile.dev/docs/getting-started/whatisnile
Nile explained in 3 minutes
**Nile is a Postgres platform that decouples storage from compute, virtualizes tenants, and supports vertical and horizontal scaling globally to ship B2B applications fast while being safe with limitless scale.** All B2B applications are multi-tenant. A tenant/customer is primarily a company, an organization, or a workspace in your product that contains a group of users. A B2B application provides services to multiple tenants. Tenant is the basic building block of all B2B applications.
## Unlimited database and virtual tenant databases
In Nile, a database is a logical concept. Our serverless compute allows us to offer a truly cost-effective, multi-tenant solution that provisions new databases rapidly. This enables Nile to provide unlimited databases, even for free tiers. Serverless compute is ideal for testing, prototyping, and supporting early customers. As customers become more active, you can seamlessly transition them to provisioned compute for enhanced security or scalability. Nile's efficiency is remarkable—a new database is provisioned in under a second. The accompanying video demonstrates a typical Nile database creation process and shows the execution of an initial use case.
## Tenant placement on both serverless or provisioned compute with 10x compute cost savings
Tenants can now be placed on different types of compute within the same database. The serverless compute is extremely cost-efficient, proving cheaper than provisioning a standard instance on RDS. Built with true multitenancy, it enables Nile to use resources more efficiently across its users. Meanwhile, highly active customers can be moved to provisioned compute. The best part? The capacity needed for this is significantly lower than for an entire database housing all customers.
## Support billions of vector embeddings across customers with 10-20x storage savings
The architecture supports vertical scaling for tenants and horizontal scaling across tenants. For vector embeddings, the total index size is divided into smaller chunks across multiple machines. Additionally, since the storage is in S3, Nile can swap a tenant’s embeddings entirely to S3 without maintaining a local cache. The indexes themselves are smaller, and multiple machines can be leveraged to build indexes in parallel. This approach provides lower latency and nearly 100% recall by reducing the search space per customer.
## Secure isolation for customer’s data and embeddings
Each tenant in this architecture functions as its own virtual database. The Postgres connections understand tenants and can route to a specific one. Tenant data isolation is enforced natively in Postgres, as it recognizes tenant boundaries without the need for Row-Level Security (RLS). Furthermore, the architecture allows tenants to be moved instantly between compute instances. Performance isolation between tenants can be achieved by relocating them to other compute instances with more capacity without any downtime.
## Branching, backups, query insights, and read replicas by tenant/customer
Since Postgres understands tenant boundaries, we can now maintain one database for all tenants while executing database operations at the tenant level. This allows us to reproduce customer issues by simply branching the specific customer's data and replaying their workload. If a customer accidentally deletes their data, backups can be restored instantly. We can create read replicas only for customers with higher workloads, saving both compute and storage resources. Moreover, we can now debug performance for specific tenants or customers, eliminating the need to treat the database as a black box.
# AWS Bedrock
Source: https://thenile.dev/docs/integrations/aws_bedrock
[Amazon Bedrock](https://aws.amazon.com/bedrock/) is a fully managed service that offers a choice of high-performing foundation models
from leading AI companies like AI21 Labs, Anthropic, Cohere, Meta, Mistral AI, Stability AI, and Amazon through a single API,
along with a broad set of capabilities you need to build generative AI applications with security, privacy, and responsible AI.
## Using AWS Bedrock with Nile
AWS Bedrock's foundation models can be used with Nile to build B2B applications using RAG (Retrieval Augmented Generation) architectures.
Below, we show a simple example of how to use AWS Bedrock's embedding models with Nile. However, keep in mind that AWS Bedrock offers a
wide range of models and capabilities. All of them can be used with Nile to build powerful AI-native applications, using similar patterns
to the one shown below.
We'll walk you through the setup steps and then explain the code line by line. The entire script is available [here](https://github.com/niledatabase/niledatabase/blob/main/examples/integrations/code_snippets/nile_aws_bedrock_quickstart.py).
### Setting Up Nile
Once you've [signed up for Nile](https://console.thenile.dev), you'll be promoted to create your first database. Go ahead and do so. You'll be redirected to the "Query Editor" page
of your new database. This is a good time to create the table we'll be using in this example:
```sql theme={null}
create table todos (
id uuid DEFAULT (gen_random_uuid()),
tenant_id uuid,
title varchar(256),
embedding vector(1024), -- Must match the dimensionality of the embedding model we'll be using
complete boolean);
```
Once you've created the table, you'll see it on the left-hand side of the screen. You'll also see the `tenants` table that is built-in to Nile.
Next, you'll want to pick up your database connection string: Navigate to the "Settings" page, select "Connections" and click "Generate credentials".
Copy the connection string and keep it in a secure location.
To use Nile in your application, you'll also need to install Psycopg2, a Python library for interacting with Postgres.
And since we'll be using vector embeddings, it helps to have pgvector's Python client installed as well.
You can install it with the following command:
```bash theme={null}
python -m pip install psycopg2-binary pgvector
```
### Setting Up AWS Bedrock
The first thing you'll need to do is to [request access to the models](https://us-west-2.console.aws.amazon.com/bedrock/home?region=us-west-2#/modelaccess)
you'll be using. AWS model availability varies by region, so make sure you select the region where your application will be deployed and
request access to models in that region.
Note that it takes some time for AWS to approve access requests, so make sure to do this well ahead of time. In this example, we'll be using `Titan Text Embeddings V2`
model from Amazon.
You'll also need to install `boto3`, which is AWS's Python SDK. You can install it with the following command:
```bash theme={null}
python -m pip install boto3
```
The simplest way to use boto3 on your local machine is using AWS profile and credentials. If you already have them configured, thats great!
Otherwise, you'll need to use AWS CLI to configure them. [AWS Boto3 documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html) explains how to do this as well as other ways you can authenticate.
### Quickstart
Now that we have everything set up, we can start writing some code (or alternatively, you can download the entire script [here](https://github.com/niledatabase/niledatabase/blob/main/examples/integrations/code_snippets/nile_aws_bedrock_quickstart.py) and follow along.
First, we'll need to import the libraries we'll be using and setting up the Boto3 Bedrock client:
```python theme={null}
import boto3
import json
import psycopg2
from pgvector.psycopg2 import register_vector
# Create a Bedrock Runtime client in the AWS Region of your choice.
client = boto3.client("bedrock-runtime", region_name="us-west-2")
model_id = "amazon.titan-embed-text-v2:0" # you can try other models as well, once you request access
```
Next, we'll set up the connection to the Nile database, register the pgvector client with the cursor, and create a tenant who will own the todo items:
```python theme={null}
conn = psycopg2.connect('postgresql://user:password@us-west-2.db.thenile.dev:5432/mydb')
conn.set_session(autocommit=True)
cur = conn.cursor()
register_vector(cur)
cur.execute("insert into tenants (name) values ('first tenant') returning id;")
tenant_id = cur.fetchone()[0]
```
Now we'll generate embeddings for a few todo items and insert them into Nile:
```python theme={null}
todo_items = [
"Center a div",
"Implement RAG-based HR chatbot",
"Add OKTA authentication to the app",
"Write a blog post about RAG with Cohere and Nile",
"Optimize a slow database query",
]
# Turn each todo item into a request to the model and convert the request to a JSON string.
# Amazing Titan Embeddings model doesn't accept batch requests, so we need to send one item at a time.
requests = [{"inputText": item} for item in todo_items]
# Call the model with each request and store the response in Nile
for item, request in zip(todo_items, requests):
json_response = client.invoke_model(body=json.dumps(request), modelId=model_id)
response = json.loads(json_response.get('body').read())
cur.execute("INSERT INTO todos (tenant_id, title, embedding) VALUES (%s, %s, %s)", (tenant_id, item, response.get('embedding')))
```
Now we'll use Nile's RAG capabilities to retrieve todo items related to a given query:
```python theme={null}
question = "Is there any work left on authentication?"
question_embedding = client.invoke_model(body=json.dumps({"inputText": question}), modelId=model_id)
question_embedding = json.loads(question_embedding.get('body').read())
# Search for the question embedding in the database
cur.execute("set nile.tenant_id = %s", (tenant_id,))
cur.execute("SELECT title, complete FROM todos ORDER BY embedding <#> %s::vector LIMIT 1", (question_embedding.get('embedding'),))
print(cur.fetchone())
```
Run the script with the following command:
```bash theme={null}
python cohere_nile_quickstart.py
```
And if everything went well, you should see the following output:
```
('Add OKTA authentication to the app')
```
Seems like a good answer to the question!
## Full application
Coming soon!
# BetterAuth
Source: https://thenile.dev/docs/integrations/betterauth
Integrate BetterAuth with Postgres
A Next.js-based authentication project using the Better-Auth plugin for organizations, adapted to work with the Nile database.
While we can use the Better-Auth organization plugin with Nile without modifications, doing so loses the tenant isolation features that Nile offers.
For the Better-Auth organization plugin to work with Nile, we need to achieve the following 4 things:
To store organizations, we must use the integrated tenants table instead of organizations, because we need a reference to this table (tenant\_id) to isolate tenants.
To store organization members, we must use the tenant\_users table instead of the members table, as this allows us to check if a user has access to a tenant using the Nile SDK. Example:
```typescript theme={null}
nile.getInstance({
tenantId: tenantId,
userId: userId,
api: {
token: token,
},
});
```
To achieve this, we use Better-Auth's optional schema definition to utilize the required tables.
To store user information, optionally, we can use the integrated users table from Nile instead of the user table used by Better-Auth. While this does not affect Nile's operation, using the users table prevents having two tables performing the same function.
To use these tables, we simply define a custom schema for the plugin in the auth.ts file.
The organization plugin has multiple foreign keys that reference different tables. To integrate it with Nile, we must sacrifice some of them because Nile has limitations on using foreign keys between shared and isolated tables. See: [tenant-sharing](http://localhost:3001/tenant-virtualization/tenant-sharing). It also has a limitation when using unique columns in the tenants table.
What do we need to sacrifice?
Use of unique for organization slugs: We replace:
```sql theme={null}
"slug" text not null unique
```
with:
```sql theme={null}
"slug" text not null
```
Although multiple rows with the same slug can now exist in the table, Better-Auth performs a check before creating a new organization, so this modification does not affect functionality but does affect data integrity.
Foreign keys: To adapt foreign keys to Nile while considering the limitations of shared tables, the user reference is moved to a table that is isolated. See images for details.
Better-Auth Plugin Table Relationships
Custom Relationships: The users table remains shared, so it will have no foreign keys.
Although Better-Auth provides a function to generate custom IDs, it currently
does not work with the plugins. To fix this, we modify the function used to
generate the ID.
To do this, we need to add the tenant\_id reference to some CRUD operations. For example, we change from:
```typescript theme={null}
where: [
{
field: "id",
value: memberId,
},
],
```
to:
```typescript theme={null}
where: [
{
field: "id",
value: memberId,
},
{
field: "organizationId",
value: organizationId,
},
],
```
Modify the role type to be compatible with Nile: In the Better-Auth plugin, organization member roles are stored as strings, whereas Nile stores them as an array. To use the roles column integrated in Nile, we modify how Better-Auth stores roles, for example, changing from "admin" to \['admin'].
## Installation
1. Run the migrations.sql in your Nile console [see](https://github.com/aris-2/better-auth-nile/blob/main/apps/demo/lib/migrations.sql).
2. Run npm install better-auth-nile.
3. Configure the environment variables.
4. Configure custom ID generation in your auth.ts to use UUID [see](https://github.com/aris-2/better-auth-nile/blob/main/apps/demo/lib/auth.ts).
5. Define a custom schema for the users table, which will allow using Nile's users table [see](https://github.com/aris-2/better-auth-nile/blob/main/apps/demo/lib/auth.ts).
# Cloudflare
Source: https://thenile.dev/docs/integrations/cloudflare
Deploy globally accelerated B2B applications with Cloudflare Workers and Hyperdrive
Cloudflare Workers enables you to deploy serverless functions or entire application backends globally, drastically reducing latency between your application and users. Combined with Cloudflare Hyperdrive's caching proxy capabilities, you can achieve exceptional database performance for your B2B applications.
## Why Nile with Cloudflare?
### Global Scale
Deploy your B2B applications worldwide using Cloudflare Workers, while Nile ensures tenant isolation, auth-scaling, and tenant insights.
### Accelerated Performance
Leverage Cloudflare Hyperdrive to accelerate database queries, making interactions lightning-fast for your users.
### Multi-Tenant Isolation
Nile's tenant-aware architecture ensures that your customers' data is securely isolated, enabling you to build scalable and compliant apps effortlessly.
### Simplified Deployment
Combine Nile's serverless Postgres with Cloudflare's edge network to streamline the deployment of modern B2B applications with less operational overhead.
### Optimized for AI Workloads
Build AI-driven, multi-tenant applications with Nile and deploy them globally with Cloudflare. This architecture minimizes latency while easily scaling to millions of embeddings.
## Quick Start Guide
1. Sign up and log in to [Nile Console](https://console.thenile.dev)
2. Create a new database
3. Create your required tables using the SQL editor
4. Get your connection string with credentials from the "Connect" tab
Clone our example repository and navigate to the Cloudflare integration:
```bash theme={null}
git clone https://github.com/niledatabase/niledatabase
cd niledatabase/examples/integrations/cloudflare/
```
This example includes:
* [Hono](https://hono.dev/): A lightweight web framework
* [Drizzle](https://orm.drizzle.team/): A TypeScript ORM
* Built-in multi-tenant todo list functionality
Create a Hyperdrive configuration using your Nile connection string:
```bash theme={null}
npx wrangler hyperdrive create your_config_name \
--connection-string="postgres://..."
```
Update your `wrangler.toml` with the Hyperdrive ID returned from the command above:
```toml theme={null}
[hyperdrive]
id = "your_hyperdrive_id"
```
Test your application locally:
```bash theme={null}
npm run dev
```
Deploy to Cloudflare Workers:
```bash theme={null}
npm run deploy
```
Your globally accelerated multi-tenant application is now live! 🚀
Now that you have your application running, you can:
* Explore the [complete example](https://github.com/niledatabase/niledatabase/tree/main/examples/integrations/cloudflare) on GitHub
* Learn more about [Nile's multi-tenant features](/tenant-virtualization/tenant-isolation)
* Check out [Cloudflare Workers documentation](https://developers.cloudflare.com/workers/)
* Join our [Discord community](https://discord.gg/8UuBB84tTy) for support
## Example Implementation
Here's a simple example of how to use Nile with Cloudflare Workers:
```typescript theme={null}
import { Hono } from 'hono';
import { drizzle } from 'drizzle-orm/neon-serverless';
import { todos } from './schema';
const app = new Hono();
app.post('/api/todos', async (c) => {
const { title } = await c.req.json();
const tenant_id = c.req.header('x-tenant-id');
const db = drizzle(c.env.HYPERDRIVE);
const [todo] = await db
.insert(todos)
.values({ title, tenant_id })
.returning();
return c.json(todo);
});
export default app;
```
# Cohere
Source: https://thenile.dev/docs/integrations/cohere
[Cohere](https://cohere.com) is an Enterprise AI platform that provides developers with world-class models, along with the supporting
platform required to deploy them securely and privately.
## Using Cohere and Nile together
Cohere's chat models, embedding models and re-ranker can be used with Nile to build B2B applications using
RAG (Retrieval Augmented Generation) architectures. Cohere is a sophisticated platform, and there are several ways you can use it with Nile to build RAG architectures.
We will demonstrate a simple example of using Cohere's embedding model with Nile, but the same principles can be applied to the other Cohere models.
We'll walk you through the setup steps and then explain the code line by line. The entire script is available [here](https://github.com/niledatabase/niledatabase/blob/main/examples/integrations/code_snippets/nile_cohere_quickstart.py).
To get started, you'll need a Cohere account and a Nile database. You can sign up for Cohere [here](https://dashboard.cohere.com/welcome/register) and for Nile [here](https://console.thenile.dev).
### Setting Up Nile
Once you've [signed up for Nile](https://console.thenile.dev), you'll be promoted to create your first database. Go ahead and do so. You'll be redirected to the "Query Editor" page
of your new database. This is a good time to create the table we'll be using in this example:
```sql theme={null}
create table todos (
id uuid DEFAULT (gen_random_uuid()),
tenant_id uuid,
title varchar(256),
embedding vector(1024),
complete boolean);
```
Once you've created the table, you'll see it on the left-hand side of the screen. You'll also see the `tenants` table that is built-in to Nile.
Next, you'll want to pick up your database connection string: Navigate to the "Settings" page, select "Connections" and click "Generate credentials".
Copy the connection string and keep it in a secure location.
To use Nile in your application, you'll also need to install Psycopg2, a Python library for interacting with Postgres.
And since we'll be using vector embeddings, it helps to have pgvector's Python client installed as well.
You can install it with the following command:
```bash theme={null}
python -m pip install psycopg2-binary pgvector
```
### Setting Up Cohere
After you've signed up for Cohere, select the "API Keys" link on the left-hand side of the screen.
Click the "Create API Key" button, copy the key and keep it in a secure location.
You can start with a more limited trial key, and upgrade to production key once you're ready to scale up.
Next, you'll want to install Cohere's Python SDK. You can do this with the following command:
```bash theme={null}
python -m pip install cohere
```
### Quickstart
We'll start with a simple example that where we use Cohere's embedding model to generate embeddings for a few todo items.
And then we'll use Nile's RAG capabilities to retrieve todo items related to a given query.
First, create a new file called `cohere_nile_quickstart.py`.
We'll start by setting up the Cohere client:
```python theme={null}
import cohere
import psycopg2
from pgvector.psycopg2 import register_vector
cohere = cohere.Client('your-api-key')
model = "embed-english-v3.0" # Replace with your favorite Cohere model
```
Next, we'll set up the connection to the Nile database, register the pgvector client with the curson, and create a tenant who will own the todo items:
```python theme={null}
conn = psycopg2.connect('postgresql://user:password@us-west-2.db.thenile.dev:5432/mydb')
conn.set_session(autocommit=True)
cur = conn.cursor()
register_vector(cur)
cur.execute("insert into tenants (name) values ('first tenant') returning id;")
tenant_id = cur.fetchone()[0]
```
Now let's use Cohere to generate embeddings for a few todo items and insert them into Nile:
```python theme={null}
todo_items = [
"Center a div",
"Implement RAG-based HR chatbot",
"Add OKTA authentication to the app",
"Write a blog post about RAG with Cohere and Nile",
"Optimize a slow database query",
]
response = cohere.embed(model=model, texts=todo_items, input_type="search_document").embeddings
for item, embedding in zip(todo_items, response):
cur.execute("INSERT INTO todos (tenant_id, title, embedding) VALUES (%s, %s, %s)", (tenant_id, item, embedding))
```
And finally, lets find the todo items that are most similar to a given question from an impatient project manager:
```python theme={null}
question = "Is there any work left on authentication?"
question_embedding = cohere.embed(model=model, texts=[question], input_type="search_query").embeddings[0]
cur.execute("set nile.tenant_id = %s", (tenant_id,))
cur.execute("SELECT title FROM todos ORDER BY embedding <#> %s::vector LIMIT 1", (question_embedding,))
print(cur.fetchone())
```
Run the script with the following command:
```bash theme={null}
python cohere_nile_quickstart.py
```
And if everything went well, you should see the following output:
```
('Add OKTA authentication to the app')
```
Seems like a good answer to the question!
## Full application
Coming soon!
# Confluent Cloud
Source: https://thenile.dev/docs/integrations/confluent
[Confluent Cloud](https://www.confluent.io/confluent-cloud/) is a cloud-native service for Apache Kafka. It provides, not only Kafka clusters but also managed
schema registry, connectors, stream processing, and more. If you are building a B2B application that requires real-time data streaming, Confluent Cloud is
a good place to use these components without having to deploy and manage them yourself.
## Using Confluent Cloud and Nile together
Confluent's Postgres sink connector allows you to stream data from Kafka to Nile. This is great for use cases such as streaming user feedback to Nile for use in RAG,
as well as for any case where your application requires real-time data from a wide variety of sources
(check out their [catalog](https://www.confluent.io/product/connectors/) for an insanely long list).
In this guide, we'll walk you through how to set up Confluent Cloud to stream data to Nile. We'll use Confluent Cloud's Datagen connector to generate
fake customer feedback data, and then stream it to Nile.
### Setting Up Nile
Start by signing up for [Nile](https://console.thenile.dev/). Once you've signed up for Nile, you'll be promoted to create your first database. Go ahead and do so.
You'll be redirected to the "Query Editor" pageof your new database. This is a good time to create the table we'll be using in this example:
```sql theme={null}
CREATE TABLE "feedback" (
"rating_id" BIGINT NOT NULL,
"user_id" INT NOT NULL,
"stars" INT NOT NULL,
"route_id" INT NOT NULL,
"rating_time" BIGINT NOT NULL,
"channel" TEXT NOT NULL,
"message" TEXT NOT NULL);
```
Once you've created the table, you'll see it on the left-hand side of the screen. You'll also see the `tenants` table that is built-in to Nile.
Next, you'll want to pick up your database connection string: Navigate to the "Settings" page, select "Connections" and click "Generate credentials".
Copy the connection string and keep it in a secure location.
### Setting Up Confluent Cloud
If you haven't already, sign up for a [Confluent Cloud account](https://confluent.cloud/signup). You'll probably want to walk through their tutorial - they will walk you
through creating a cluster, a topic, and a datagen source connector.
### Quickstart
We'll assume that you've already created a cluster. In your cluster, you'll want to create a new topic. You can do this by selecting "Topics" from the left-hand side, and
clicking on "Add topic". Call the topic `feedback`, and accept the default settings.
#### Creating the source connector
Next, you'll want to create your source connector. You can do this by selecting "Connectors" from the left-hand side, search for "Sample Data" connector and click on it.
Click on "additional configuration", and configure it by following the wizard and using the following settings:
* Select "feedback" as the topic.
* Select an API key. If you don't already have one, you can create a new API key on this page.
* Select "JSON\_SR" as the value for "output record value format".
* Select "Rating" as the schema.
After this, you can accept the defaults and a connector will be created. You can click on it to see metrics and logs. You can also go to the "feedback" topic and see the data
flowing into it.
#### Creating the sink connector
Now we'll set up the sink connector, to send data from Kafka to Nile. In the connectors page, search for Postgres and make sure you click on the "Postgres Sink" connector.
Configure it by following the wizard and using the following settings:
* Select "feedback" as the topic.
* Select an API key. You can generate a new one, or use the one you created for the source connector.
* Now enter the details of your Nile database. You have all the information in the connection string you generated in a previous step. Or you can use the Settings page in
Nile console to get this information.
After this, you can accept the defaults and a connector will be created. You can click on it to see metrics and logs.
Go back to Nile Console, find the "feedback" table in the query editor and run a query:
```sql theme={null}
SELECT * FROM "feedback" limit 10;
```
You should see the data flowing into Nile.
Congratulations! You've now set up a real-time data pipeline using Confluent Cloud, Kafka, and Nile. You are all ready to start building your B2B application with Nile.
# Serverless Multi-tenant Backend with AWS Lambda
Source: https://thenile.dev/docs/integrations/lambda
In this tutorial, you will learn how to use Nile's tenant virtualization from a serverless application, using a todo list application example.
We'll use Drizzle as the ORM to interact with the database, Express as the web framework, NodeJS as the runtime and Serverless Framework for deployment.
## 1. Create a database
1. Sign up for an invite to [Nile](https://thenile.dev) if you don't have one already
2. You should see a welcome message. Click on "Lets get started"
3. Give your workspace and database names, or you can accept the default auto-generated names. In order to complete this quickstart in a browser, make sure you select to "Use Token in Browser".
## 2. Create a table
After you created a database, you will land in Nile's query editor. Since our application requires a table for storing all the "todos" this is a good time to create one:
```sql theme={null}
CREATE TABLE IF NOT EXISTS "todos" (
"id" uuid DEFAULT gen_random_uuid(),
"tenant_id" uuid,
"title" varchar(256),
"complete" boolean,
CONSTRAINT todos_tenant_id_id PRIMARY KEY("tenant_id","id")
);
```
You will see the new table in the panel on the left side of the screen, and you can expand it to view the columns.
See the `tenant_id` column? By specifying this column, You are making the table **tenant aware**. The rows in it will belong to specific tenants. If you leave it out, the table is considered shared, more on this later.
## 3. Get credentials
In the left-hand menu, click on "Settings" and then select "Connection".
Click on the Postgres button, then click "Generate Credentials" on the top right corner. Copy the connection string - it should now contain the credentials we just generated.
## 4. Set the environment
Enough GUI for now. Let's get to some code.
If you haven't cloned this repository yet, now will be an excellent time to do so.
```bash theme={null}
git clone https://github.com/niledatabase/niledatabase
cd niledatabase/examples/serverless/lambda-drizzle
```
Rename `.env.example` to `.env`, and update it with the connection string you just copied from Nile Console. Make sure you don't include the word "psql". It should look something like this:
```bash theme={null}
DATABASE_URL=postgres://018b778a-30df-7cdd-b55c-2f9664db39f3:ff3fb983-683c-4616-bbbc-519d8ddbbce5@db.thenile.dev:5432/gwen_db
```
Install dependencies:
```bash theme={null}
npm install
```
**Optional:** You can select a region for deploying this example by editing `serverless.yml` and changing the `region` property.
And if you haven't yet, install the Serverless Framework: `npm install -g serverless`
## 5. Deployment
In order to deploy the example, run the following command:
```bash theme={null}
serverless deploy
```
After running deploy, you should see output similar to:
```bash theme={null}
Deploying serverless-node-drizzle to stage dev (us-east-2)
✔ Service deployed to stack serverless-node-drizzle-dev (93s)
endpoint: ANY - https://z2fmc4ux34.execute-api.us-west-2.amazonaws.com
functions:
api: serverless-node-drizzle-dev-api (424 kB)
```
Now you can use `curl` to explore the APIs. Here are a few examples:
```bash theme={null}
# create a tenant
curl --location --request POST 'localhost:3001/api/tenants' \
--header 'Content-Type: application/json' \
--data-raw '{"name":"my first customer", "id":"108124a5-2e34-418a-9735-b93082e9fbf2"}'
# get tenants
curl -X GET 'http://localhost:3001/api/tenants'
# create a todo (don't forget to use a read tenant-id in the URL)
curl -X POST \
'http://localhost:3001/api/tenants/108124a5-2e34-418a-9735-b93082e9fbf2/todos' \
--header 'Content-Type: application/json' \
--data-raw '{"title": "feed the cat", "complete": false}'
# list todos for tenant (don't forget to use a read tenant-id in the URL)
curl -X GET \
'http://localhost:3001/api/tenants/108124a5-2e34-418a-9735-b93082e9fbf2/todos'
# list todos for all tenants
curl -X GET \
'http://localhost:3001/insecure/all_todos'
```
## 6. Check the data in Nile
Go back to the Nile query editor and see the data you created from the app.
```sql theme={null}
SELECT tenants.name, title, complete
FROM todos join tenants on tenants.id = todos.tenant_id;
```
You should see all the todos you created, and the tenants they belong to.
## 7. How does it work?
In this section we'll focus on the serverless aspects. If you want to learn more about how to use Nile with Drizzle,
check out our [Drizzle getting started guide](https://www.thenile.dev/docs/getting-started/languages/drizzle).
In order to deploy the backend to AWS Lambda, we use the [Serverless Framework](https://www.serverless.com/).
The framework handles most of the configuration and deployment for us, and required minimal changes to the code.
The changes we did make:
* Wrapped the Express app with a Serverless handler.
* Made sure the database connections is initialized outside the handler with no top level await
* Added a `serverless.yml` file to the root of the project. This file contains the configuration for the Serverless Framework.
Lets go over these one by one:
### 7.1. Wrapping the Express app
In the example, we used `serverless-http` NodeJS module, which wraps an Express app with a Serverless handler.
Using it is very straightforward:
```javascript theme={null}
import serverless from 'serverless-http';
const app = express();
// all the application logic goes here - handlers, middleware, etc
export const handler = serverless(app);
```
### 7.2. Initializing the database connection
The `serverless-http` wrapping above is almost enough to get the application running on AWS Lambda.
However, we need to make sure the database connection is initialized before the handler is called and remains intact between handler executions.
We are initializing the connection in the `db.js` file, and exporting the connection object.
```javascript theme={null}
const client = new Client({
connectionString: process.env.DATABASE_URL,
});
export const db = drizzle(client.connect(), { logger: true });
```
This means that the connection is initialized when the file is imported, during the initial "cold start" of the serverless application,
and remains initialized as long as the virtual machine is running. Since in AWS Lambda, the virtual machine is reused between executions,
this minimizes the connection overhead for all practical appluications.
### 7.3. The `serverless.yml` file
Now we just need to configure the deployment and our serverless application is ready to go.
The `serverless.yml` file contains the configuration for the Serverless Framework and it has 4 important sections.
The first is the general configuration. We set the name of the service, the framework version, and most important - the use of `.env` file for environment variables.
```yaml theme={null}
service: lambda-drizzle
frameworkVersion: '3'
useDotenv: true
```
Next, there's the provider section where we configure AWS itself.
We set the runtime to NodeJS 18, the region to `us-west-2` and include the database connection string in the environment.
The region is important - you want to run your serverless application in the same region as your database. In this case - `us-west-2`.
```yaml theme={null}
provider:
name: aws
runtime: nodejs18.x
region: us-west-2
environment:
DATABASE_URL: ${env:DATABASE_URL}
```
The next section is the functions. Here we connect the API routing to the application handlers.
In our case, it is pretty simple - all the routes are handled by the Express app, which then routes them to the correct handler.
```yaml theme={null}
functions:
api:
handler: app.handler
events:
- httpApi: '*'
```
And last, Serverless Framework has a collection of plugins that extend its basic capabilities. In this case, because we are using
Typescript with ES modules, we need to use the `serverless-esbuild` plugin.
```yaml theme={null}
plugins:
- serverless-esbuild
```
# LlamaIndex
Source: https://thenile.dev/docs/integrations/llama_index
[LlamaIndex](https://llamaindex.ai/) is a framework for building context-augmented generative AI applications with LLMs.
It provides a wide range of functionality including data connectors, index building, query engines, agents, workflows and observability.
Making it easy to build powerful RAG applications.
## Using LlamaIndex and Nile together
LlamaIndex can be used with Nile to build RAG (Retrieval Augmented Generation) architectures.
You'll use LLamaIndex to simply and orchestrate the different steps in your RAG workflows, and Nile to store and query data and embeddings.
In this example, we'll show how to chat with a sales transcript in just a few lines of code, using LlamaIndex's high-level interface and its integration with Nile and OpenAI.
We'll walk you through the setup steps and then explain the code line by line. The entire Python script is available
[here](https://github.com/niledatabase/niledatabase/blob/main/examples/integrations/code_snippets/nile_llamaindex_quickstart.py)
or in [iPython/Jupyter Notebook](https://github.com/niledatabase/niledatabase/blob/main/examples/integrations/code_snippets/nile_llamaindex_quickstart.ipynb).
### Setting Up Nile
Start by signing up for [Nile](https://console.thenile.dev/?utm_campaign=partnerlaunch\&utm_source=nilewebsite\&utm_medium=docs). Once you've signed up for Nile, you'll be promoted to create your
first database. Go ahead and do so. You'll be redirected to the "Query Editor" page of your new database. You can see the built-in `tenants` table on the left-hand side.
From there, click on "Home" (top icon on the left menu), click on "generate credentials" and copy the resulting connection string. You will need it in a sec.
### Setting Up LlamaIndex
LlamaIndex is a Python library, so you'll need to set up a Python environment with the necessary dependencies.
We recommend using [venv](https://docs.python.org/3/library/venv.html) to create a virtual environment.
This step is optional, but it will help you manage your dependencies and avoid conflicts.
```bash theme={null}
python3 -m venv llama-env
source llama-env/bin/activate
```
Once you've activated your virtual environment, you can install the necessary dependencies - LlamaIndex and the Nile Vector Store:
```bash theme={null}
pip install llama-index llama-index-vector-stores-nile
```
### Setting up the data
In this example, we'll chat with sales transcript of two different companies. Download the transcripts to `./data` directory.
```bash theme={null}
mkdir -p data/
wget "https://raw.githubusercontent.com/niledatabase/niledatabase/main/examples/ai/sales_insight/data/transcripts/nexiv-solutions__0_transcript.txt" -O "data/nexiv-solutions__0_transcript.txt"
wget "https://raw.githubusercontent.com/niledatabase/niledatabase/main/examples/ai/sales_insight/data/transcripts/modamart__0_transcript.txt" -O "data/modamart__0_transcript.txt"
```
### Setting up the OpenAI API key
This quickstart uses OpenAI's API to generate embeddings. So grab your [OpenAI API key](https://platform.openai.com/api-keys) and set it as an environment variable:
```bash theme={null}
export OPENAI_API_KEY="your-openai-api-key"
```
## Quickstart
Open a file named `nile_llamaindex_quickstart.py` and start by importing the necessary dependencies (or follow along with the script mentioned above):
```python theme={null}
import logging
logging.basicConfig(level=logging.INFO)
from llama_index.core import SimpleDirectoryReader, StorageContext
from llama_index.core import VectorStoreIndex
from llama_index.core.vector_stores import (
MetadataFilter,
MetadataFilters,
FilterOperator,
)
from llama_index.vector_stores.nile import NileVectorStore, IndexType
```
### Setting up the NileVectorStore
Next, create a NileVectorStore instance:
```python theme={null}
vector_store = NileVectorStore(
service_url="postgresql://user:password@us-west-2.db.thenile.dev:5432/niledb",
table_name="test_table",
tenant_aware=True,
num_dimensions=1536,
)
```
Note that in addition to the usual parameters like URL and dimensions, we also set `tenant_aware=True`. This is because we want to isolate the documents for each tenant in our vector store.
🔥 NileVectorStore supports both tenant-aware vector stores, that isolates the documents for each tenant and a regular store which is typically used for shared data that all tenants can access.
Below, we'll demonstrate the tenant-aware vector store.
### Loading and transforming the data
With all this in place, we'll load the data for the sales transcripts. We'll use LlamaIndex's `SimpleDirectoryReader` to load the documents. Because we want to update the documents with the tenant
metadata after loading, we'll use a separate reader for each tenant.
```python theme={null}
reader = SimpleDirectoryReader(input_files=["nexiv-solutions__0_transcript.txt"])
documents_nexiv = reader.load_data()
reader = SimpleDirectoryReader(input_files=["modamart__0_transcript.txt"])
documents_modamart = reader.load_data()
```
We are going to create two Nile tenants and the add the tenant ID of each to the document metadata. We are also adding some additional metadata like a custom document ID and a category. This metadata can be used for filtering documents during the retrieval process.
Of course, in your own application, you could also load documents for existing tenants and add any metadata information you find useful.
```python theme={null}
tenant_id_nexiv = str(vector_store.create_tenant("nexiv-solutions"))
tenant_id_modamart = str(vector_store.create_tenant("modamart"))
# Add the tenant id to the metadata
for i, doc in enumerate(documents_nexiv, start=1):
doc.metadata["tenant_id"] = tenant_id_nexiv
doc.metadata[
"category"
] = "IT" # We will use this to apply additional filters in a later example
doc.id_ = f"nexiv_doc_id_{i}" # We are also setting a custom id, this is optional but can be useful
for i, doc in enumerate(documents_modamart, start=1):
doc.metadata["tenant_id"] = tenant_id_modamart
doc.metadata["category"] = "Retail"
doc.id_ = f"modamart_doc_id_{i}"
```
We are loading all documents to the same `VectorStoreIndex`. Since we created a tenant-aware `NileVectorStore` when we set things up,
Nile will correctly use the `tenant_id` field in the metadata to isolate them. Loading documents without `tenant_id` to a tenant-aware store will throw a `ValueException`.
```python theme={null}
storage_context = StorageContext.from_defaults(vector_store=vector_store)
index = VectorStoreIndex.from_documents(
documents_nexiv + documents_modamart,
storage_context=storage_context,
show_progress=True,
)
```
### Chatting with the documents
Now that we have our vector embeddings stored in Nile, we can build a query engine for each tenant and chat with the documents:
```python theme={null}
nexiv_query_engine = index.as_query_engine(
similarity_top_k=3,
vector_store_kwargs={
"tenant_id": str(tenant_id_nexiv),
},
)
print(nexiv_query_engine.query("What were the customer pain points?"))
modamart_query_engine = index.as_query_engine(
similarity_top_k=3,
vector_store_kwargs={
"tenant_id": str(tenant_id_modamart),
},
)
print(modamart_query_engine.query("What were the customer pain points?"))
```
And run the script:
```bash theme={null}
python nile_llamaindex_quickstart.py
```
Nexiv is an IT company and Modamart is a retail company. You can see that the query engine for each tenant returns an answer relevant to the tenant.
Thats it! You've now built a (small)RAG application with LlamaIndex and Nile.
The
[Python script](https://github.com/niledatabase/niledatabase/blob/main/examples/integrations/code_snippets/nile_llamaindex_quickstart.py)
and [iPython/Jupyter Notebook](https://github.com/niledatabase/niledatabase/blob/main/examples/integrations/code_snippets/nile_llamaindex_quickstart.ipynb) include all the code for this quickstart,
as well as few additional examples - for example, how to use metadata filters to farther restrict the search results, or how to delete documents from the vector store.
## Full application
Ready to build something amazing? Check out our [TaskGenius example](https://github.com/niledatabase/niledatabase/tree/main/examples/ai/local_llama_task_genius).
The README includes step by step instructions on how to run the application locally.
Lets go over a few of the code highlights:
### Use of LlamaIndex with FastAPI
The example is a full-stack application with a FastAPI back-end and a React front-end.
When we initialize FastAPI, we create an instance of our `AIUtils` class, which is responsible for interfacing with the Nile vector store and the OpenAI API.
```python theme={null}
@asynccontextmanager
async def lifespan(app: FastAPI):
# Initialize AIUtils - it is a singleton, so we are just saving on initialization time
AIUtils()
yield
# Cleanup
ai_utils = None
app = FastAPI(lifespan=lifespan)
```
Then, in the handler for the `POST /api/todos` endpoint, we use the `AIUtils` instance to generate a time estimate for the todo item, and to store the embedding in the Nile vector store.
```python theme={null}
@app.post("/api/todos")
async def create_todo(todo: Todo, request: Request, session = Depends(get_tenant_session)):
ai_utils = AIUtils() # get an instance of AIUtils
todo.tenant_id = get_tenant_id();
todo.id = str(uuid4())
estimate = ai_utils.ai_estimate(todo.title, todo.tenant_id)
todo.estimate = estimate
ai_utils.store_embedding(todo)
session.add(todo)
session.commit()
return todo
```
### Using LlamaIndex with Ollama
If you look at the `AIUtils` class, you'll see that it is very similar to the simple quickstart example earlier. Except we use Ollama instead of OpenAI.
We initialize the vector store and the index in the `__init__` method:
```python theme={null}
# Initialize settings and vector store once
Settings.embed_model = OllamaEmbedding(model_name="nomic-embed-text")
Settings.llm = Ollama(model="llama3.2", request_timeout=360.0)
self.vector_store = NileVectorStore(
service_url=os.getenv("DATABASE_URL"),
table_name="todos_embedding",
tenant_aware=True,
num_dimensions=768
)
self.index = VectorStoreIndex.from_vector_store(self.vector_store)
```
Then to store the embedding in the Nile vector store, we do exactly what we did in the quickstart example - enrich the todo item with the tenant ID and insert it into the index:
```python theme={null}
document = Document(
text=f"{todo.title} is estimated to take {todo.estimate} to complete",
id_=str(todo.id),
metadata={"tenant_id": str(todo.tenant_id)}
)
```
To get an estimate, we create a query engine for the tenant and use it to query the index, just like we did in the quickstart example:
```python theme={null}
query_engine = self.index.as_query_engine(vector_store_kwargs={
"tenant_id": str(tenant_id),
})
response = query_engine.query(
f'you are an amazing project manager. I need to {text}. How long do you think this will take? '
f'respond with just the estimate, no yapping.'
)
```
### Using FastAPI with Nile for tenant isolation
If you look at the `POST /api/todos` handler, you'll see that we get all the todos for a tenant without needing to do any filtering.
This is because the `get_tenant_session` function returns a session for the tenant database:
```python theme={null}
@app.get("/api/todos")
async def get_todos(session = Depends(get_tenant_session)):
results = session.exec(select(Todo.id, Todo.tenant_id,Todo.title, Todo.estimate, Todo.complete)).all()
return results
```
`get_tenant_session` is implemented in `db.py` and it is a wrapper around the `get_session` function that sets the `nile.tenant_id` and `nile.use_id` context.
```python theme={null}
def get_tenant_session():
session = Session(bind=engine)
try:
tenant_id = get_tenant_id()
user_id = get_user_id()
session.execute(text(f"SET LOCAL nile.tenant_id='{tenant_id}';"))
# This will raise an error if user_id doesn't exist or doesn't have access to the tenant DB.
session.execute(text(f"SET LOCAL nile.user_id='{user_id}';"))
yield session
```
The tenant\_id and user\_id are set in the context by our custom FastAPI middleware. It extracts the tenant ID from the request headers and user token from the cookies.
You can see it in `tenant_middleware.py`.
```python theme={null}
class TenantAwareMiddleware:
def __init__(self, app):
self.app = app
async def __call__(self, scope, receive, send):
headers = Headers(scope=scope)
maybe_tenant_id = headers.get("X-Tenant-Id")
maybe_set_context(maybe_tenant_id, tenant_id)
request = Request(scope)
token = request.cookies.get("access_token")
maybe_user_id = get_user_id_from_valid_token(token)
maybe_set_context(maybe_user_id, user_id)
await self.app(scope, receive, send)
```
### Summary
This example shows how to use LlamaIndex with Nile to build a RAG application. It demonstrates how to store and query documents in a tenant-aware vector store, and how to use
metadata filters to further restrict the search results. It also shows how to use FastAPI with Nile to build a full-stack application with tenant isolation.
# Modal
Source: https://thenile.dev/docs/integrations/modal
[Modal](https://modal.com) is a serverless compute platform that supports a wide range of applications, from web services to AI models.
Its user-friendly approach allows developers to write standard Python code with minimal additional annotations and deploy it to the cloud.
Importantly, Modal seamlessly integrates with any Python Postgres client, making it fully compatible with Nile.
By combining Modal's serverless compute capabilities with Nile's serverless database, developers can create entirely serverless AI applications, streamlining their development process and infrastructure management.
## Using Modal and Nile together
We'll demonstrate a simple example of using Modal's serverless compute capabilities with Nile's serverless database. All the setup steps and line-by-line explanation are shown below.
The entire script is available [here](https://github.com/niledatabase/niledatabase/blob/main/examples/integrations/code_snippets/nile_modal_quickstart.py).
To get started, you'll need a Modal account and a Nile database. You can sign up for Modal [here](https://modal.com/signup) and for Nile [here](https://console.thenile.dev).
### Setting Up Nile
Once you've [signed up for Nile](https://console.thenile.dev), you'll be promoted to create your first database. Go ahead and do so. You'll be redirected to the "Query Editor" page
of your new database. This is a good time to create the table we'll be using in this example:
```sql theme={null}
create table todos (
id uuid DEFAULT (gen_random_uuid()),
tenant_id uuid,
title varchar(256),
complete boolean);
```
Once you've created the table, you'll see it on the left-hand side of the screen. You'll also see the `tenants` table that is built-in to Nile.
Next, you'll want to pick up your database connection string: Navigate to the "Settings" page, select "Connections" and click "Generate credentials".
Copy the connection string and keep it in a secure location. You'll need it to connect to your database from Modal.
### Setting Up Modal
Once you've signed up for Modal, you'll be ready to start building your application.
In order to connect from Modal to your Nile database, you'll need to create a secret in Modal.
Click on the "Secrets" tab and then "Create Secret". We recommend choosing "Custom" secret type (although you can use "Postgres" as well).
In the secret key, enter `DATABASE_URL`. In the secret value, enter the connection string you picked up in the previous step.
Make sure you leave out the `psql` and replace `postgres://` with `postgresql://`. When you save the secret, call it `my-nile-secret`.
Next, you'll want to install modal and set it up:
```bash theme={null}
pip install modal
modal setup
```
### Quickstart
Lets try creating and running a simple example. Just one function that runs on Modal and queries the Nile database.
First, create a new file called `nile_modal_quickstart.py`.
We'll start by setting up the Modal app and function. You can see that we are using a pre-existing image that Modal provides,
and we are installing `psycopg2` which is a Postgres client for Python. Then we create the modal app.
```python theme={null}
import os
import modal
import psycopg2
image = modal.Image.debian_slim(python_version="3.10").pip_install(
"psycopg2-binary",
)
app = modal.App("quickstart-nile-modal")
```
Next, we'll create a function that will run on Modal. You can see that we are initializing it with our image and the secret we created earlier.
```python theme={null}
@app.function(image=image, secrets=[modal.Secret.from_name("my-nile-secret")])
def nile_quickstart():
conn = psycopg2.connect(os.getenv("DATABASE_URL"))
conn.set_session(autocommit=True)
cur = conn.cursor()
tenant_id = create_tenant("My first tenant", cur)
create_todo(tenant_id, "My first todo", cur)
print(f"Created todo for first tenant. Showing all todos:\n {show_todos(cur)}\n\n")
tenant_id = create_tenant("Another tenant", cur)
create_todo(tenant_id, "My second todo", cur)
print(f"Created second tenant and their todo. Showing all todos:\n {show_todos(cur)}\n\n")
cur.execute("SET nile.tenant_id = %s", (tenant_id,) )
# No need to change the query, since we are connected to a specific virtual tenant database
print(f"Connected to virtual tenant database for tenant {tenant_id}. Showing all todos:\n {show_todos(cur)}\n")
cur.close()
conn.close()
```
This function uses the connection string from the Modal secret to connect to the Nile database.
It then creates two tenants and two todos. Each todo will be associated with one of the tenants.
`show_todos` is a helper function that will query simply return all todos in the database with their respective tenant name.
You can see that once we connect to the virtual tenant database, the same query will only show the todos for the tenant we connected to.
Create the helper functions `create_tenant`, `create_todo` and `show_todos` and you're ready to run the function:
```python theme={null}
def create_tenant(name: str, cur: psycopg2.extensions.cursor):
cur.execute("INSERT INTO tenants (name) VALUES (%s) RETURNING id;", (name,))
return cur.fetchone()[0]
def create_todo(tenant_id: int, title: str, cur: psycopg2.extensions.cursor):
cur.execute("INSERT INTO todos (tenant_id, title) VALUES (%s, %s);", (tenant_id, title))
return True
def show_todos(cur: psycopg2.extensions.cursor):
# Note the lack of filter on the query - we always show all todos in the database
cur.execute("SELECT tenants.name, title FROM todos join tenants on tenants.id = todos.tenant_id;")
return cur.fetchall()
```
### Running on Modal
To run the function on Modal, you can use the following command:
```bash theme={null}
modal run nile_modal_quickstart.py
```
## Full application
Now that you've successfully deployed your first function to Modal and used your first Nile database, let's put them together in a full application.
We'll create a Sales Assistant application that allows you to embed and summarize sales conversations.
The entire application - frontend, backend and LLM - all run on Modal. All the data and embeddings are stored in Nile.
Get Started with Sales Assistant
# Netlify
Source: https://thenile.dev/docs/integrations/netlify
Netlify is a platform for delivering modern web applications. It supports a wide range of frameworks and provides a comprehensive set of tools for building, deploying,
and managing web apps.
## Using Netlify and Nile together
Netlify supports deployment of various web frameworks and of serverless functions.
Nile's serverless database is a great fit for frameworks like NextJS and for Netlify serverless functions. Together they can be used to build B2B applications that are entirely serverless.
With Nile's tenant virtualization, you can store private data and embeddings for each of your customers in a secure and scalable manner.
In this example, we'll show you how we deploy a NextJS application to Netlify that uses Nile as a database.
### Setting Up Nile
Start by signing up for [Nile](https://console.thenile.dev/). Once you've signed up for Nile, you'll be promoted to create your first database. Go ahead and do so.
You'll be redirected to the "Query Editor" pageof your new database. This is a good time to create the table we'll be using in this example:
```sql theme={null}
CREATE TABLE todos (
id uuid DEFAULT (gen_random_uuid()),
tenant_id uuid,
title varchar(256),
estimate varchar(256),
embedding vector(768), -- must match the embedding model dimension
complete boolean);
```
Once you've created the table, you'll see it on the left-hand side of the screen. You'll also see the `tenants` table that is built-in to Nile.
**Note:** You can also start by setting up a new Netlify project and enabling the Nile integration from there. If you choose this route, make sure you still create the `todos` table.
### Setting Up Netlify
If you don't already have a Netlify account, go ahead and create one [here](https://www.netlify.com/). Pretty simple.
In your Netlify dashboard, navigate to "Extensions", choose "Nile" and click on the Tile.
You'll see a page that looks like this:
Click on the "Connect" button. You'll be redirected to Nile, where you can log in and authorize Netlify to create databases and database credentials on your behalf.
Once this is done, you'll be redirected back to Netlify. Time to create your project!
### Quickstart
You'll want to start by forking [Nile's github repo](https://github.com/niledatabase/niledatabase). You can do this by clicking the "Fork" button in the top right corner of the screen.
Now that you have your own fork, you have the option to import it to Netlify and configure the deployment using the UI or the CLI. Let's do it via the UI.
In Netlfiy dashboard, navigate to "Sites", click on "Add new site", choose "Import existing project" and pick "github". Authorize Netlify access to your GitHub account and select your fork.
Back on Netlfiy, you'll be asked to choose a name for your project. We chose `nile-netlify-todo`, but you can choose anything you like.
When choosing "Site to Deploy", make sure you choose the "main" branch and the "todo-nextjs" site.
This project needs a few environment variables to be set (in addition to the Nile database connection variables that you'll get automatically from the extension). You can set them here:
```bash theme={null}
# the URL of this example + where the api routes are located
NEXT_PUBLIC_APP_URL=https://nile-netlify-todo.netlify.app/api
# For automatic time estimates of todos - replace with your own
AI_API_KEY=xHOdh...
AI_BASE_URL=https://api.fireworks.ai/inference/v1
AI_MODEL=accounts/fireworks/models/llama-v3p1-405b-instruct
```
Use your own Netlify project name in the `NEXT_PUBLIC_APP_URL` variable.
And use whatever OpenAI-compatible inference API you like in the `AI_API_KEY`, `AI_BASE_URL` and `AI_MODEL` variables.
Once you set this, click on "Deploy site".
In the Site screen, click on "Site configuration". Scroll down to "Nile" (or select it on the left hand menu), and pick the database you created earlier.
Click "Save". This will automatically populate Nile's connection information in your environment variables. Then check on the "Deploys" tab to see the build in progress.
You will need to re-build in order to pick up any new environment variables you set.
Once the the build is complete, you'll see "Open Production Deploy" button on top. Click on it to open your app.
### Congratulations!
You've now deployed a serverless backend that uses Nile as the database and Netlify as the serverless compute platform.
Feel free to explore the app (or you can try [the live demo version](https://nile-netlify-todo.netlify.app/) of it). You can sign up as a new user, create a new tenant, create todos for that tenant, and even use the embedding search functionality.
You will see all these users, tenants and todos stored in your Nile database.
You can explore them in [Nile Console](https://console.thenile.dev/) or via your favorite Postgres client.
# NextAuth Authentication with Nile
Source: https://thenile.dev/docs/integrations/nextauth
This example shows how to use [NextAuth](https://next-auth.js.org/) with Nile's tenant management and isolation. NextAuth is a popular and easy to use authentication
library for Next.js applications. Using NextAuth with Nile as the database gives you access to passwordless authentication, session-based identity, and most important - tenant isolation.
Properly configured, Nile will automatically validate that a user has access to the tenant when executing queries on behalf of a user.
## Getting Started
### 1. Create a new database
Sign up for an invite to [Nile](https://thenile.dev) if you don't have one already and choose "Yes, let's get started". Follow the prompts to create a new workspace and a database.
### 2. Create todo table
After you created a database, you will land in Nile's query editor. Since our application requires a table for storing all the "todos" this is a good time to create one:
```sql theme={null}
create table todos (id uuid DEFAULT (gen_random_uuid()), tenant_id uuid, title varchar(256), complete boolean);
```
If all went well, you'll see the new table in the panel on the left hand side of the query editor. You can also see Nile's built-in tenant table next to it.
### 3. Create tables for NextAuth data model
```sql theme={null}
-- These are the extra tables NextAuth requires in your database to support
CREATE TABLE IF NOT EXISTS "accounts" (
"userId" uuid NOT NULL,
"type" text NOT NULL,
"provider" text NOT NULL,
"providerAccountId" text NOT NULL,
"refresh_token" text,
"access_token" text,
"expires_at" integer,
"token_type" text,
"scope" text,
"id_token" text,
"session_state" text,
CONSTRAINT account_provider_providerAccountId_pk PRIMARY KEY("provider","providerAccountId"),
CONSTRAINT account_userId_users_id_fk FOREIGN KEY ("userId") REFERENCES users.users("id") ON DELETE cascade ON UPDATE no action
);
CREATE TABLE IF NOT EXISTS "sessions" (
"sessionToken" text PRIMARY KEY NOT NULL,
"userId" uuid NOT NULL,
"expires" timestamp NOT NULL,
CONSTRAINT "session_userId_users_id_fk" FOREIGN KEY ("userId") REFERENCES users.users("id") ON DELETE cascade ON UPDATE no action
);
CREATE TABLE IF NOT EXISTS "verification_token" (
"identifier" text NOT NULL,
"token" text NOT NULL,
"expires" timestamp NOT NULL,
CONSTRAINT verificationToken_identifier_token_pk PRIMARY KEY("identifier","token")
);
-- users table is built-in to Nile, but we need to extend it for NextAuth
alter table users.users add column "email_verified" timestamp;
```
Those should also show up on the left panel.
### 4. Getting credentials
In the left-hand menu, click on "Settings" and then select "Credentials". Generate credentials and keep them somewhere safe. These give you access to the database.
### 5. Setting the environment
If you haven't cloned this project yet, now will be an excellent time to do so. Since it uses NextJS, we can use `create-next-app` for this:
```bash theme={null}
npx create-next-app -e https://github.com/niledatabase/niledatabase/tree/main/examples/user_management/NextAuth todo-nextauth
cd nile-todo
```
Rename `.env.local.example` to `.env.local`, and update it with your workspace and database name.
*(Your workspace and database name are displayed in the header of the Nile dashboard.)*
Also fill in the username and password with the credentials you picked up in the previous step.
Our example includes passwordless email and Github OAuth authentication.
To use either method, you'll want to fill in the appropriate section of the environment file.
You can refer to NextAuth getting started guides with [email](https://authjs.dev/getting-started/providers/email-tutorial)
or [oauth](https://authjs.dev/getting-started/providers/oauth-tutorial) for more details.
The resulting env fileshould look something like this:
```bash theme={null}
# Random string for encryption
NEXTAUTH_SECRET="random string. You should change this to something very random."
# For passwordless authentication, we need SMTP credentials
SMTP_USER=postmaster@youremaildomain.com
SMTP_PASSWORD=supersecret
SMTP_HOST=smtp.youremailhost.com
SMTP_PORT=587
EMAIL_FROM=someone@youremaildomain.com
# For Github OAuth, we need the client ID and secret we got when we registered with Github:
GITHUB_ID=Iv1.aa6f0f2bfa21b677
GITHUB_SECRET=b5b35282aa04bee94f31ccfaa44ed8c5e00fd2a9b2
# Private (backend) env vars for connecting to Nile database
NILE_DB_HOST = "db.thenile.dev"
NILE_USER =
NILE_PASSWORD =
NILE_DATABASE =
# Client (public) env vars
# the URL of this example + where the api routes are located
# Use this to instantiate Nile context for client-side components
NEXT_PUBLIC_BASE_PATH=http://localhost:3000/api
NEXT_PUBLIC_WORKSPACE=
NEXT_PUBLIC_DATABASE=
```
Install dependencies with `npm install`.
### 6. Running the app
```bash theme={null}
npm run dev
```
Open [http://localhost:3000](http://localhost:3000) with your browser to see the result.
If all went well, your browser should show you the first page in the app, asking you to login or sign up.
After you sign up as a user of this example app, you'll be able to see this user by going back to Nile Console and running `select * from users.users` in the query editor.
Login with the new user, and you can create a new tenant and add tasks for the tenant. You can see the changes in your Nile database by running
```sql theme={null}
select name, title, complete from
tenants join todos on tenants.id=todos.tenant_id
```
## How it works
### Setting up NextAuth
NextAuth is a very flexible authentication library that supports a wide range of authentication methods and providers. It is very easy to configure and use.
We set it up in [`app/api/auth/[...nextauth]/route.js`](https://github.com/niledatabase/niledatabase/blob/main/examples/user_management/NextAuth/app/api/auth/%5B...nextauth%5D/route.ts):
```javascript theme={null}
export const authOptions: NextAuthOptions = {
adapter: NileAdapter(pool),
callbacks: {...}
providers: [
GithubProvider({
clientId: process.env.GITHUB_ID,
clientSecret: process.env.GITHUB_SECRET,
}),
Email({...}),
],
}
const handler = NextAuth(authOptions)
export { handler as GET, handler as POST }
```
The `NileAdapter` is a custom adapter that implements the NextAuth adapter interface. It uses the Nile database to store user information and sessions.
This route handles all calls to `/api/auth/*` and provides the following endpoints:
* `/api/auth/signin` - handles sign in requests
* `/api/auth/signout` - handles sign out requests
* `/api/auth/session` - returns the current session
* `/api/auth/providers` - returns a list of configured providers
* `/api/auth/callback/*` - handles callbacks from authentication providers
* `/api/auth/csrf` - returns a CSRF token
Which we then use in our application.
### Using NextAuth for Login / Logout
NextAuth SDK provides `signIn()` method that we call and it handles the login flow for us. We use it in [`app/pages.tsx`](https://github.com/niledatabase/niledatabase/blob/main/examples/user_management/NextAuth/app/page.tsx).
```javascript theme={null}
import { signIn } from 'next-auth/react';
signIn('github', { callbackUrl: '/tenants' })}>
Sign in with Github
;
```
As you can see, this is very easy to use. We just need to provide the provider name and the URL where we want the user to land after authenticating.
NextAuth will handle the rest.
Similarly, to provide a logout link in [`app/tenants/page.tsx`](https://github.com/niledatabase/niledatabase/blob/main/examples/user_management/NextAuth/app/tenants/page.tsx)
we link to the signout endpoint provided by NextAuth:
```javascript theme={null}
(Logout)
```
### Using NextAuth for identity and access
NextAuth provides a `useSession()` hook that we can use to get the current session.
In order to use it with Nile's tenant isolation, we refer to it when retrieving a connection to Nile's tenant databases in [`/lib/NileServer.ts`](https://github.com/niledatabase/niledatabase/blob/main/examples/user_management/NextAuth/lib/NileServer.ts)
This guarantees that all queries executed on behalf of the user will be executed in the context of the tenant the user is currently logged in to.
Nile will also respond with errors if the user is not authorized to access the tenant.
```javascript theme={null}
export async function configureNile(tenantId: string | null | undefined) {
console.log("configureNile", tenantId);
const session = await getServerSession(authOptions);
console.log(session);
return nile.getInstance({
tenantId: tenantId,
//@ts-ignore
userId: session?.user?.id,
});
}
```
## Adding new authentication providers
NextAuth supports a wide range of authentication providers. You can see the full list [here](https://next-auth.js.org/providers/).
If you want to modify this example to use a different provider, you can do so by first modifying the NextAuth configuration
in [`app/api/auth/[...nextauth]/route.js`](https://github.com/niledatabase/niledatabase/blob/main/examples/user_management/NextAuth/app/api/auth/%5B...nextauth%5D/route.ts).
Import the provider you want to choose and add a section to the `authOptions` object.
Then, you'll need to modify the UI to use the new provider. For example, if you want to use Google OAuth, you'll need to add a button to the UI that calls `signIn("google")`.
Thats it. NextAuth will handle the rest and you will have a new authentication method for your multi-tenant application.
# Pipedream
Source: https://thenile.dev/docs/integrations/pipedream
Automate customer workflow
By combining Nile's tenant-aware database with [Pipedream](https://pipedream.com/)'s workflow automation, you can build these integrations quickly and reliably, without getting lost in integration code. Pipedream has hundreds of integrations (over 2400 per their website), so you can truly connect your tenants to anything.
## Integration Types
Pipedream supports two types of integrations with Nile:
* Nile as a trigger: Pipedream can run a query every few minutes and initialize a workflow based on the results. Perhaps updating your CRM when a tenant modifies their account in your application or email tenants after the create their 10th user.
* Nile as an action: Use this when you want to create a tenant, a user or run a query in Nile as a result of a trigger. Maybe you want to create a tenant when your sales team creates a new customer in Salesforce, or update tenant's status based on payment events from Stripe.
## Setting Up Your First Workflow
Lets start with a very simple example that you can try without having access to Stripe or Salesforce accounts. To try this out, you'll need credentials for a Nile database and a Pipedream account.
In Pipedream, create a workspace or choose an existing one. Then click on new to create a new workflow.
We'll use HTTP trigger, since those are simple to test. Click on Add trigger and then select New HTTP / Webhook Requests.
Selecting a trigger
You can leave all the defaults as is and just click Save and continue. Click on Generate test event. This will send a simple HTTP request and Pipedream will show you all the available information about HTTP events. All these fields will be available in the workflow.
HTTP event fields
Now lets add Nile action to the workflow. Click on the + icon in the flow diagram, search for Nile and pick Execute Query action.
In the Action configuration, under Nile Database Account click on Connect Account. This will redirect you to Nile, where you'll be able to give Pipedream access to your workspaces and databases. This is control-plane access. Pipedream will still require credentials to access any specific database.
Once you gave Pipedream access to Nile, you'll be able to pick your workspace and database from the dropdown menus. Then, provide username and password credentials to the database you picked.
Nile configuration
Finally, we tell Pipedream what query to run. Lets say that we simply want to log the HTTP request in a shared table. We can use the following query. Note how we use the values exported by the trigger in our query:
```sql theme={null}
INSERT INTO pipe (client_ip, url) VALUES
('{{steps.trigger.event.client_ip}}', '{{steps.trigger.event.url}}')
```
Before testing the pipeline, you'll want to make sure the pipe table exists, otherwise the insert statement will fail:
```sql theme={null}
CREATE TABLE pipe (client_ip VARCHAR(16), url VARCHAR(1024));
```
As the last step, click on Test and then check the pipe table in your database. If all went well, you'll see something like this:
```sql theme={null}
stripe_demo=> SELECT * FROM pipe;
client_ip | url
--------------+------------------------------------------
75.3.243.203 | https://eoly6egu9oxmmq2.m.pipedream.net/
75.3.243.203 | https://eoly6egu9oxmmq2.m.pipedream.net/
(2 rows)
```
Hopefully this was fun. While HTTP triggers may not be what you are after, you can use the exact same steps to build all kinds of integrations. Simply pick a trigger, run a test to examine the available fields, and then add actions that use these fields. Each workflow can have as many triggers and actions as you need, so you can get quite fancy!
## Real-World Integration Examples
Let's look at a few useful B2B workflows that you can build with Nile and Pipedream:
### New Customer (From product):
1. Trigger: Nile
2. Event: New tenant
3. Actions: Add new customer in Hubspot
4. Actions: Update team via Slack
### New Customer (From sales team):
1. Trigger: Hubspot
2. Event: New Deal in Stage. Configure the source to only trigger on "won" deals.
3. Actions: Create tenant in Nile
4. Actions: Send welcome email via SendGrid
### Support Ticket
1. Trigger: Zendesk
2. Event: New ticket
3. Actions: Query Nile for more information about tenant - subscription type, number of users, etc.
4. Actions: Update tenant metadata in Zendesk
5. Actions: If ticket is high priority, send Slack message to customer success team
### Subscription Cancelled
1. Trigger: Stripe
2. Event: Cancelled subscription
3. Actions: Update tenant status in Nile
4. Actions: Notify customer success team via Slack
## Tips
When building these integrations, keep these tips in mind:
* Always Preserve Tenant Context: Make sure you're passing tenant IDs through your entire workflow, especially when executing queries againt tenant data in Nile. Pipedream makes this easy by exporting event variables in each step.
* Handle Errors Gracefully: Things will go wrong. Set up proper error handling and notifications.
# Build SaaS with paid subscriptions - using Nile and Stripe
Source: https://thenile.dev/docs/integrations/stripe
In this quick tutorial, you will learn how to extend Nile's built-in tables with a Stripe subscription data, and how to use Stripe's API to manage subscriptions.
## 1. Create a database
1. Sign up for an invite to [Nile](https://thenile.dev) if you don't have one already
2. You should see a welcome message. Click on "Lets get started"
3. Give your workspace and database names, or you can accept the default auto-generated names. In order to complete this quickstart in a browser, make sure you select to “Use Token in Browser”.
## 2. Extend the tenant table
After you created a database, you will land in Nile's query editor. Stripe integration requires storing customer and subscription IDs.
For that, we'll extend the built-in `tenants` table:
```sql theme={null}
alter table tenants add column stripe_customer_id text;
alter table tenants add column stripe_subscription_id text;
alter table tenants add column tenant_tier varchar(16) default 'free';
```
If all went well, you'll see the new columns in the panel on the left side of the query editor.
## 3. Get credentials
In the left-hand menu, click on "Settings" and then select "credentials". Generate credentials and keep them somewhere safe. These give you access to the database.
Also, copy and save the workspace and database names. You'll need them later.
## 4. Sign up to Stripe and Create a Product
1. Sign up for a [Stripe account](https://dashboard.stripe.com/register)
2. Go to the [Stripe Dashboard](https://dashboard.stripe.com/test/dashboard) and click on "Developers" -> "API Keys", on the upper right corner.
3. Copy the "Secret Key" and "Publishable Key" and keep them somewhere safe. You'll need them later.
4. On the left menu, click on "more +" and select "Product Catalog". Add a new product, and make sure you select "Recurring" as the pricing model. You can use the default values for the rest of the fields.
## 4. Set the environment
Enough GUI for now. Let's get to some code.
If you haven't cloned this repository yet, now will be an excellent time to do so.
```bash theme={null}
git clone https://github.com/niledatabase/niledatabase
cd niledatabase/examples/quickstart/nextjs
```
Rename `.env.local.example` to `.env.local`, and update it with your workspace and database name.
*(Your workspace and database name are displayed in the header of the Nile dashboard.)*
Also fill in the username and password with the credentials you picked up in the previous step.
It should look something like this:
```bash theme={null}
# Client (public) env vars
# the URL of this example + where the api routes are located
# Use this to instantiate Nile context for client-side components
NEXT_PUBLIC_BASE_PATH=http://localhost:3000
NEXT_PUBLIC_WORKSPACE=todoapp_demo
NEXT_PUBLIC_DATABASE=stripe_demo_db
# Private env vars that should never show up in the browser
# These are used by the server to connect to Nile database
NILE_DB_HOST = "db.thenile.dev"
NILE_USER = "018ad484-0d52-7274-8639-057814be60c3"
NILE_PASSWORD = "0d11b8e5-fbbc-4639-be44-8ab72947ec5b"
STRIPE_SECRET_KEY = "sk_test_51Nn2AgJ5..."
# The URL of the Nile API
# Use this to instantiate Nile Server context for server-side use of the "api" SDK
NEXT_PUBLIC_NILE_API=https://api.thenile.dev
# Uncomment if you want to try Google Auth
# AUTH_TYPE=google
```
Install dependencies:
```bash theme={null}
npm install
```
### 5. Run the application
```bash theme={null}
npm run dev
```
💡 Note: This example only works with Node 18 and above. You can check the version with `node -v`.
Open [http://localhost:3000](http://localhost:3000) with your browser to see the result.
If all went well, your browser should show you the first page in the app, asking you to login or sign up.
After you sign up as a user of this example app, you'll be able to see this user by going back to Nile Console and looking at the users table
```sql theme={null}
select * from users;
```
Login with the new user, and you can create a new tenant by clicking on the "Create Tenant" button. This will take you into the example billing page.
You can play around with this - upgrade and downgrade the tenant (use Stripe's test credit card number `4242 4242 4242 4242`),
and see the changes in the database:
```sql theme={null}
select * from tenants
```
Will show you the tenant tier and the Stripe customer and subscription IDs. Note that only paid tiers have a subscription ID.
## 7. How does it work?
There are a few moving pieces here, so let's break it down.
This example uses NextJS `app router`, so the application landing page is in `app/page.tsx`.
We'll start here and go over the code for managing tenant subscriptions.
### 7.1. Configuring Nile SDK
All the signup and login methods eventually route users to `/tenants`. You'll find the code in `app/tenants/page.tsx`.
The first thing we do in this page is to configure Nile SDK for the current user:
```typescript theme={null}
nile.withContext({ headers });
```
This method configures a shared instance of `nile` that is used throughout the application.
This instance is a singleton, which you get by calling Nile SDK's `Server` method, which we are doing in '@/lib/NileServer'
This is also where all the environment variables we've set earlier are being used, so lets take a look at this file:
```typescript theme={null}
const nile = Nile({
basePath: String(process.env.NEXT_PUBLIC_NILE_API),
});
```
The `basePath` configuration is the URL that `nile` methods will call. This component calls Nile APIs directly, and therefore we set `basePath` to `NEXT_PUBLIC_NILE_API`.
So every page, route and function in our app can use the same `nile` instance to access Nile APIs and DB.
But, we need to make sure we are using the right user and tenant context.
So we call `configureNile` and pass in the cookies and the tenant ID.
After this point, we can use `nile` to access the database and APIs, and it will use the right user and tenant context.
### 7.2 Creating a Paid subscription
When tenants are created, we initially create them in the `free` tier. This is done in `app/tenants/tenant-actions.tsx`:
```typescript theme={null}
const createTenantResponse = await nile.tenants.create(
{
name: tenantName,
},
true,
);
```
Note that we don't pass in a `tier` parameter. This is because we want to create the tenant in the default tier, which is `free`.
We handled that when we extended `tenants` table with a column for the tenant tier:
```sql theme={null}
alter table tenants add column tenant_tier varchar(16) default 'free';
```
After the tenant is created, we can upgrade it to a paid tier. This is initiated in `app/tenants/[tenantid]/billing/page.tsx` :
```jsx theme={null}
```
When a user clicks the button, it triggets `createCheckoutSession` which is a NextJS server action implemented in `app/tenants/[tenantid]/billing/checkout-actions.tsx`:
```typescript theme={null}
export async function createCheckoutSession(formData: FormData) {
const tenantid = formData.get('tenantid')?.toString();
const prices = await stripe.prices.list(); // (1)
const price = prices.data[0].id;
const session = await stripe.checkout.sessions.create({
// (2)
billing_address_collection: 'auto',
line_items: [
{
price: price,
quantity: 1,
},
],
mode: 'subscription',
success_url:
process.env.NEXT_PUBLIC_BASE_PATH +
`/api/checkout-success?session_id={CHECKOUT_SESSION_ID}&tenant_id=${tenantid}`,
cancel_url:
process.env.NEXT_PUBLIC_BASE_PATH + `/tenants/${tenantid}/billing`,
});
const url: string = session.url || '/'; // (3)
redirect(url);
}
```
What this does is:
1. Call Stripe to get the price ID for the product (alternatively you can get it in Stripe's dashboard and use an environment variable for this).
2. Call Stripe's API to create a checkout session. This is where we pass in the success and cancel URLs.
The success URL is a NextJS route that we'll see in a second. Note that we pass in the tenant ID as a query parameter in the success URL. This lets us identify the tenant when Stripe calls the success URL, and upgrade the correct tenant.
The cancel URL simply takes the user back to the billing page.
3. Redirect the user to Stripe's checkout page. It's URL is part of the session we just created..
### 7.3 Handling Stripe's checkout success
When the user completes the checkout process, Stripe will call the success URL we passed in earlier. This is a NextJS route, implemented in `app/api/checkout-success/page.tsx`.
This is where we upgrade the tenant to a paid tier and store his customer and subscription references in our `tenants` table:
```typescript theme={null}
console.log('checkout-success called');
const stripe = new Stripe(process.env.STRIPE_SECRET_KEY || '');
const searchParams = req.nextUrl.searchParams;
const tenantId = searchParams.get('tenant_id')?.toString();
const session_id = searchParams.get('session_id')?.toString();
let location: string;
const checkoutSession = await stripe.checkout.sessions.retrieve(session_id); // (1)
// Here we are getting a connection to a specific tenant database
const tenantNile = await nile.withContext({ headers, tenantId }); // (2)
// Store the Stripe customer ID and subscription in the database
const resp = await tenantNile.query(
'update tenants set (stripe_customer_id = $1, stripe_subscription_id = $2, tenant_tier = $3',
[checkoutSession.customer, checkoutSession.subscription, 'basic'],
);
revalidatePath('/tenants');
return respond('/tenants/' + tenantId + '/billing');
```
1. First we call Stripe to get the checkout session. This is where we get the customer ID and subscription ID.
2. Then we get an instance of the tenantDB using the tenant ID, which we cleverly asked Stripe to provide when calling this route.
3. Finally, we update the tenant's row in the `tenants` table with the customer and subscription IDs, and upgrade the tenant to the `basic` tier.
### 7.4 Managing subscriptions
We use Stripe's customer dashboard to let users manage their subscriptions.
To direct users to the dashboard, we use Stripe's `customer_portal` API. This is implemented in `app/tenants/[tenantid]/billing/checkout-actions.tsx`:
```typescript theme={null}
export async function redirectToStripePortal(formData: FormData) {
const tenantId = formData.get('tenantid')?.toString();
const tenantNile = await nile.withContext({ headers, tenantId });
const resp = await tenantNile.tenants.list();
const customerId = resp[0].stripe_customer_id;
const returnUrl =
process.env.NEXT_PUBLIC_BASE_PATH + `/tenants/${tenantId}/billing`;
const portalSession = await stripe.billingPortal.sessions.create({
customer: customerId,
return_url: returnUrl,
});
redirect(portalSession.url);
}
```
In order to call Stripe's API, we need the customer ID. We get it from the `tenants` table, and then call Stripe's API to create a portal session.
We then redirect the user to the portal session URL, which takes the user to the customer dashboard.
### 7.5 Downgrading a tenant
A tenant who is subscribed to the "basic" tier can downgrade to the "free" tier. This is implemented in `app/tenants/[tenantid]/billing/checkout-actions.tsx`:
```typescript theme={null}
export async function cancelSubscription(formData: FormData) {
const tenantid = formData.get('tenantid')?.toString();
const tenantNile = await nile.withContext({ headers, tenantId });
const resp = await tenantNile.tenants.list();
const subscriptionId = resp[0].stripe_subscription_id;
try {
await stripe.subscriptions.cancel(subscriptionId);
// if we got here, subscription was cancelled successfully, lets downgrade the tenant tier too
await tenantNile.query(
"update tenants set (tenant_tier = 'free' AND stripe_subscription_id = NULL)",
);
} catch (e) {
console.error(e);
return { message: 'Failed to cancel subscription' };
}
revalidatePath('/tenants/' + tenantid + '/billing');
redirect('/tenants/' + tenantid + '/billing');
}
```
This is similar to the previous example, except that we use the tenant's subscription ID to call Stripe's `cancel` API and cancel the subscription.
If this step succeeds, we downgrade the tenant in our database.
## 8. What's next?
This example is a good starting point for introducing subscriptions, tiered plans and billing for your SaaS application and your tenants.
You have also learned how to extend Nile's built-in tables with additional fields and use them in your application.
Next steps could be to add more tiers, display past payments in the customer dashboard, or to add a webhook to handle Stripe's subscription events.
You can learn more about Nile's tenant virtualization features in the following tutorials:
* [Tenant management](/tenant-virtualization/tenant-management)
* [Tenant isolation](/tenant-virtualization/tenant-isolation)
And you can explore Nile's JS SDK in the [SDK reference](/auth/sdk-reference/javascript/overview).
# Vercel
Source: https://thenile.dev/docs/integrations/vercel
## Using Vercel and Nile together
[Vercel](https://vercel.com) provides the developer tools and cloud infrastructure to build, scale, and secure high performance web applications.
Nile's NextJS examples such as the [AI-native Todo App](https://www.thenile.dev/templates/AI-Native%20multi-tenant%20SaaS%20with%20Nile%20and%20NextJS),
[KnowledgeAI](https://www.thenile.dev/templates/KnowledgeAI%20-%20PDF%20search%20assistant%20for%20your%20organization)
and [Code Assistant](https://www.thenile.dev/templates/Autonomous%20Code%20Assistant%20-%20Code%20more%2C%20type%20less) are all written in NextJS and deployed on Vercel.
In this guide we'll walk you through using Vercel's dashboard to create a NextJS project integrated with Nile database. We'll use Nile's NextJS quickstart as an example.
If you don't already have a Vercel account, go ahead and create one [here](https://vercel.com/).
### Create a database
1. Login to your Vercel workspace and click on the **storage** tab and then on **Create Database**
2. From the **Marketplace Database Providers** select **Nile Database** and click on **continue**.
3. Choose a region and a price tier (price tier can be changed later, the region is fixed).
4. Edit the auto-generated database name, if you wish, and click on **create**.
Thats it. You have a database. You should see environment variables with the database connection info as well as other example snippets.
We are planning to use this database for an example todo list application. We need a table for storing all these todos.
To create the table, click on **Open in Nile**. This will open a new tab with a query editor connected to your brand new database.
Copy-paste this snippet and click **Run**:
```sql theme={null}
CREATE TABLE todos (
id uuid DEFAULT (gen_random_uuid()),
tenant_id uuid,
title varchar(256),
estimate varchar(256),
embedding vector(768), -- must match the embedding model dimension
complete boolean);
```
You'll see the new table in the left side menu, as well as the built-in tenants table.
### Deploying Nile's NextJS Quickstart
1. Start by forking [Nile's github repo](https://github.com/niledatabase/niledatabase). You can do this by clicking the "Fork" button in the top right corner of the screen.
2. Back in Vercel's dashboard, click on **Add New\*** to create a new project. The "Import project" section will show the repository you just cloned - click on **Import**.
3. You'll be redirected to the "Configure project" screen. There are few things we need to change:
* **Project name**: Give your project a meaningful name. For example `nile-vercel-nextjs`.
* **Root Directory**: Set the root directory to `/examples/quickstart/nextjs`
4. Click "Deploy".
You can wait for the deployment to complete or you can cancel the deployment immediately since we still need to connect a database.
### Connecting Nile Database to a Project
1. In the project dashboard, select the **storage** tab. You'll see the database that you created in a previous step. Click on **connect**.
You can use the same database for all environments, or use separate databases for development and production. You can create additional Nile databases from this screen, if needed, with the **create database** button.
2. Once you connected a database to the project, the project's environment variables will populate with the database connection details.
3. Re-deploy the project for the new configuration to take effect.
You've now deployed a smart todo list app, written in Next.js, that uses Nile as the database and Vercel as the serverless compute platform.
### Create a database from the CLI
So far, you've explored two methods for creating Nile databases from Vercel: via the Project Storage tab and the Team Storage tab.
If you prefer using the command line, you can create a Nile database with the following command:
```bash theme={null}
vc i nile
```
Follow the prompts to select a region and tier. Once the database is created, use the command:
```bash theme={null}
vc env pull .env.local
```
This pulls the environment variables, including the database connection details, directly into your local environment.
# Creating a table in Postgres
Source: https://thenile.dev/docs/postgres/createtable
A PostgreSQL table is a structured storage object that organizes data into rows and columns. Each row represents a single record, while each column represents a specific attribute of the data. Tables use primary keys to identify rows uniquely and can establish relationships with other tables through foreign keys. They enforce data integrity and consistency using constraints.
Creating a table involves defining the table name and its columns with their respective data types.
### Example:
```sql theme={null}
CREATE TABLE employees (
employee_id SERIAL PRIMARY KEY,
first_name VARCHAR(50),
last_name VARCHAR(50),
email VARCHAR(100)
);
```
### Detailed Explanation:
* `CREATE TABLE employees`: This command tells PostgreSQL to create a new table named `employees`.
* `employee_id SERIAL PRIMARY KEY`:
* `employee_id` is the name of the column.
* `SERIAL` is a special data type that automatically generates a unique identifier for each row (an auto-incrementing integer).
* `PRIMARY KEY` means this column uniquely identifies each row in the table.
* `first_name VARCHAR(50)`:
* `first_name` is the name of the column.
* `VARCHAR(50)` specifies that this column can store up to 50 characters.
* `last_name VARCHAR(50)` and `email VARCHAR(100)` follow the same pattern.
## Adding Columns
To add a new column to an existing table, you use the `ALTER TABLE` statement.
### Example:
```sql theme={null}
ALTER TABLE employees
ADD COLUMN date_of_birth DATE;
```
### Detailed Explanation:
* `ALTER TABLE employees`: This command specifies the table (`employees`) to be modified.
* `ADD COLUMN date_of_birth DATE`:
* `ADD COLUMN` specifies that a new column is being added.
* `date_of_birth` is the name of the new column.
* `DATE` is the data type for the new column, which stores date values.
## Defining Primary Keys
A primary key is a column or a set of columns that uniquely identifies each row in a table. It ensures that each row has a unique identifier.
### Adding a Primary Key During Table Creation:
```sql theme={null}
CREATE TABLE employees (
employee_id SERIAL PRIMARY KEY,
first_name VARCHAR(50),
last_name VARCHAR(50),
email VARCHAR(100)
);
```
### Adding a Primary Key to an Existing Table:
If the table already exists without a primary key, you can add it using the `ALTER TABLE` statement:
```sql theme={null}
ALTER TABLE employees
ADD PRIMARY KEY (employee_id);
```
### Detailed Explanation:
* `ADD PRIMARY KEY (employee_id)`:
* `ADD PRIMARY KEY` specifies the addition of a primary key.
* `(employee_id)` indicates that the `employee_id` column is being set as the primary key.
## Defining Foreign Keys
A foreign key is a column or a set of columns that establishes a link between the data in two tables. It ensures referential integrity by enforcing a relationship between the columns of two tables.
### Example:
First, create the `departments` table:
```sql theme={null}
CREATE TABLE departments (
department_id SERIAL PRIMARY KEY,
department_name VARCHAR(50)
);
```
### Adding a Foreign Key to the `employees` Table:
1. Add the `department_id` column to the `employees` table:
```sql theme={null}
ALTER TABLE employees
ADD COLUMN department_id INTEGER;
```
1. Define the foreign key relationship:
```sql theme={null}
ALTER TABLE employees
ADD CONSTRAINT fk_department
FOREIGN KEY (department_id) REFERENCES departments(department_id);
```
### Detailed Explanation:
* `ALTER TABLE employees`: This command specifies the table (`employees`) to be modified.
* `ADD COLUMN department_id INTEGER`:
* `ADD COLUMN` specifies that a new column is being added.
* `department_id` is the name of the new column.
* `INTEGER` is the data type for the new column, which stores integer values.
* `ADD CONSTRAINT fk_department FOREIGN KEY (department_id) REFERENCES departments(department_id)`:
* `ADD CONSTRAINT fk_department` names the new constraint `fk_department`.
* `FOREIGN KEY (department_id)` specifies that the `department_id` column in the `employees` table is a foreign key.
* `REFERENCES departments(department_id)` indicates that this foreign key references the `department_id` column in the `departments` table.
### Full Example:
Combining all the above concepts, here's how to create both tables with the necessary columns and constraints:
```sql theme={null}
-- Create departments table
CREATE TABLE departments (
department_id SERIAL PRIMARY KEY,
department_name VARCHAR(50)
);
-- Create employees table
CREATE TABLE employees (
employee_id SERIAL PRIMARY KEY,
first_name VARCHAR(50),
last_name VARCHAR(50),
email VARCHAR(100),
date_of_birth DATE,
department_id INTEGER,
CONSTRAINT fk_department
FOREIGN KEY (department_id)
REFERENCES departments(department_id)
);
```
This script creates two tables: `departments` and `employees`, with the `employees` table having a foreign key that references the `departments` table. Each table and column is defined with appropriate data types and constraints to ensure data integrity and relationships.
# Array types in Postgres
Source: https://thenile.dev/docs/postgres/datatype/arrays
An array in PostgreSQL is a collection of elements with the same data type. It’s like a flexible container that can hold multiple values. Arrays can be one-dimensional or multi-dimensional, allowing you to represent complex data structures within a single column.
## Declaration of Array Types
To create an array column in a table, use the following syntax:
```sql theme={null}
CREATE TABLE sal_emp (
name text,
pay_by_quarter integer[],
schedule text[][]
);
```
In this example:
* `name` is a regular text column.
* `pay_by_quarter` is a one-dimensional array of integers, representing an employee’s salary by quarter.
* `schedule` is a two-dimensional array of text, representing an employee’s weekly schedule.
You can also use the `ARRAY` keyword for one-dimensional arrays:
```sql theme={null}
CREATE TABLE tictactoe (
squares integer[3][3]
);
```
Note that the array data type is named by appending square brackets (`[]`) to the data type name of the array elements. [The declared size or number of dimensions doesn’t affect runtime behavior; it’s mainly for documentation purposes1](https://www.postgresql.org/docs/current/arrays.html).
## Array Value Input
To write an array value as a literal constant, enclose the element values within curly braces and separate them by commas:
```sql theme={null}
-- Creating an array of integers
SELECT '{10, 20, 30}'::integer[];
-- Creating an array of text
SELECT '{"Monday", "Tuesday", "Wednesday"}'::text[];
```
## Accessing Array Elements
You can access individual elements of an array using the `[index]` syntax. The first element has an index of one:
```sql theme={null}
-- Accessing the second element of an integer array
SELECT pay_by_quarter[2] FROM sal_emp;
```
## Modifying Arrays
Arrays are mutable. Use the `ARRAY[...]` constructor to create or modify arrays:
```sql theme={null}
-- Adding a new value to an existing integer array
UPDATE sal_emp
SET pay_by_quarter = pay_by_quarter || ARRAY[40]
WHERE name = 'John Doe';
```
## Searching in Arrays
You can search for specific values within an array using operators like `ANY` or `ALL`:
```sql theme={null}
-- Finding employees with a salary greater than 30 in any quarter
SELECT name
FROM sal_emp
WHERE 30 < ANY (pay_by_quarter);
```
## Array Functions and Operators
PostgreSQL provides a rich set of functions and operators for working with arrays. Some useful ones include:
* `array_length(arr, dim)`: Returns the length of the array along the specified dimension.
* `unnest(arr)`: Expands an array into a set of rows.
* `array_agg(expr)`: Aggregates values into an array.
* `@>` and `<@`: Tests if an array contains another array or value.
For more detailed information, consult the official [PostgreSQL documentation](https://www.postgresql.org/docs/current/arrays.html). If you have further questions, feel free to ask us on our [Discord](https://discord.gg/8UuBB84tTy)!
# Character types in Postgres
Source: https://thenile.dev/docs/postgres/datatype/character
When working with strings in PostgreSQL, understanding the available character types is crucial. As a new user, let’s explore these types in more detail.
### 1. `character varying(n)` (or `varchar(n)`)
* **Description:**
* Variable-length character type.
* Can store strings up to `n` characters (not bytes) in length.
* Excess characters beyond the specified length result in an error unless they are spaces (in which case the string is truncated).
* **Example:**
* Suppose we want to create a `users` table to store usernames. First, let’s create the table:
```sql theme={null}
CREATE TABLE users (
user_id SERIAL PRIMARY KEY,
username VARCHAR(50) -- Define the maximum length (e.g., 50 characters)
);
```
* Now we can insert a username:
```sql theme={null}
INSERT INTO users (username) VALUES ('alice');
```
* **Use Case:**
* Use `varchar` when you need flexibility in string length, such as for user-generated content.
### 2. `character(n)` (or `char(n)`)
* **Description:**
* Fixed-length, blank-padded character type.
* Similar to `character varying(n)` but always pads with spaces.
* **Example:**
* Let’s create an `employees` table to store employee IDs:
```sql theme={null}
CREATE TABLE employees (
employee_id CHAR(5) -- Define the fixed length (e.g., 5 characters)
);
```
* Insert an employee ID:
```sql theme={null}
INSERT INTO employees (employee_id) VALUES ('E001');
```
* **Use Case:**
* Use `char` when you require fixed-length strings (e.g., employee IDs).
### 3. `bpchar` (unlimited length, blank-trimmed)
* **Description:**
* Similar to `char`, but without a specified length.
* Accepts strings of any length, and trailing spaces are insignificant.
* **Example:**
* Create a `products` table for storing product codes:
```sql theme={null}
CREATE TABLE products (
product_code BPCHAR -- No specific length defined
);
```
* Insert a product code:
```sql theme={null}
INSERT INTO products (product_code) VALUES ('P12345');
```
* **Use Case:**
* Use `bpchar` when you want to trim trailing spaces.
### 4. `text` (variable unlimited length)
* **Description:**
* PostgreSQL’s native string data type.
* Stores strings of any length.
* **Example:**
* Let’s create an `articles` table for storing article content:
```sql theme={null}
CREATE TABLE articles (
article_id SERIAL PRIMARY KEY,
content TEXT -- No length restriction
);
```
* Insert article content:
```sql theme={null}
INSERT INTO articles (content) VALUES ('Lorem ipsum dolor sit amet...');
```
* **Use Case:**
* Use `text` for general-purpose text storage.
### Operations and Considerations:
* All character types support standard string functions (e.g., `LENGTH`, `SUBSTRING`, `CONCAT`).
* Performance considerations:
* `text` is the most flexible but may have slightly slower indexing.
* Fixed-length types (`char`, `bpchar`) are faster for exact-length lookups.
* As a new user, start with `text` or `character varying` unless you have specific requirements. Feel free to experiment with different types based on your application needs!
For more detailed information, consult the official [PostgreSQL documentation](https://www.postgresql.org/docs/current/datatype-character.html). If you have further questions, feel free to ask us on our [Discord](https://discord.gg/8UuBB84tTy)!
# Date and Time Types in Postgres
Source: https://thenile.dev/docs/postgres/datatype/date
When working with dates and time in PostgreSQL, understanding the available date/time types is useful. As a Postgres user, let’s explore these types in more detail.
## 1. Date Data Type
* **Date**: Represents a date value (e.g., birthdays, deadlines).
* Stored as 4 bytes.
* Range: 4713 BC to 5874897 AD.
* Example:
```sql theme={null}
CREATE TABLE events (
event_id serial PRIMARY KEY,
event_date date
);
INSERT INTO events (event_date) VALUES ('2024-07-10');
```
## 2. Time Data Type
* **Time**: Represents a time-of-day value (e.g., appointment times).
* Stored as 8 bytes.
* Range: 00:00:00 to 24:00:00.
* Example:
```sql theme={null}
CREATE TABLE appointments (
appt_id serial PRIMARY KEY,
appt_time time
);
INSERT INTO appointments (appt_time) VALUES ('14:30:00');
```
## 3. Timestamp Data Type
* **Timestamp**: Represents a combined date and time value (e.g., event timestamps).
* Stored as 8 bytes.
* Range: 4713 BC to 294276 AD.
* Example:
```sql theme={null}
CREATE TABLE log_entries (
log_id serial PRIMARY KEY,
log_timestamp timestamp
);
INSERT INTO log_entries (log_timestamp) VALUES ('2024-07-10 15:45:00');
```
## 4. Timestamp with Time Zone Data Type
* **Timestamp with Time Zone (timestamptz)**: Includes time zone information.
* Stored as 8 bytes.
* Example:
```sql theme={null}
CREATE TABLE meetings (
meeting_id serial PRIMARY KEY,
meeting_time timestamptz
);
INSERT INTO meetings (meeting_time) VALUES ('2024-07-10 10:00:00-07');
```
## 5. Interval Data Type
* **Interval**: Represents a duration of time (e.g., event duration).
* Stored as 16 bytes.
* Example:
```sql theme={null}
CREATE TABLE tasks (
task_id serial PRIMARY KEY,
task_duration interval
);
INSERT INTO tasks (task_duration) VALUES ('2 days 3 hours');
```
Remember to adapt these examples to your specific use case. Happy querying!
For more detailed information, consult the official [PostgreSQL documentation](https://www.postgresql.org/docs/current/datatype-datetime.html). If you have further questions, feel free to ask us on our [Discord](https://discord.gg/8UuBB84tTy)!
# JSON in Postgres
Source: https://thenile.dev/docs/postgres/datatype/json
The `jsonb` data type in PostgreSQL allows you to store JSON (JavaScript Object Notation) data in a binary format.
***
### **Key Features of `jsonb`:**
* **Efficient Storage**: `jsonb` is stored in a binary representation, making it more compact than `json`.
* **Better Performance**: `jsonb` allows indexing, making it faster for querying and retrieval.
* **Flexibility**: It supports a variety of operations like indexing, full-text search, and partial updates.
* **Order Independent**: Key order is not preserved (unlike `json`), and duplicate keys are removed.
***
### **How to Use the `jsonb` data type with `tenant_id` in PostgreSQL**
### **1. Creating a table with `jsonb` and `tenant_id` column**
```sql theme={null}
CREATE TABLE products (
id INT,
tenant_id UUID NOT NULL,
name TEXT NOT NULL,
details jsonb,
CONSTRAINT PK_products PRIMARY KEY(tenant_id,id)
);
```
In this example:
* `id`: An integer primary key.
* `tenant_id`: A `UUID` type column for tenant isolation.
* `name`: A `TEXT` field for product name.
* `details`: A `jsonb` field that will store additional product details like specifications, availability, etc.
### **2. Inserting data into a `jsonb` column with `tenant_id`**
You can insert JSON data into the `jsonb` column, along with a `tenant_id`, using standard SQL `INSERT`:
```sql theme={null}
INSERT INTO products (id, tenant_id, name, details)
VALUES (1, '550e8400-e29b-41d4-a716-446655440000', 'Smartphone', '{"brand": "XYZ", "model": "A100", "price": 699.99}');
```
Here, the `tenant_id` is a UUID that ties the data to a specific tenant.
### **3. Querying `jsonb` data by `tenant_id`**
### **Accessing specific keys for a specific tenant**
You can access specific keys in the `jsonb` column and filter by `tenant_id`:
```sql theme={null}
SELECT details->'brand' FROM products
WHERE tenant_id = '550e8400-e29b-41d4-a716-446655440000' AND name = 'Smartphone';
```
This will return the brand for a given tenant’s product.
### **Using the `@>` operator for containment**
The `@>` operator checks if one JSON object contains another JSON object. It’s useful for filtering rows by specific keys/values for a tenant.
```sql theme={null}
SELECT * FROM products
WHERE tenant_id = '550e8400-e29b-41d4-a716-446655440000'
AND details @> '{"brand": "XYZ"}';
```
This finds all products with brand "XYZ" for a specific tenant.
### **4. Updating `jsonb` data for a tenant**
You can use the `jsonb_set` function to update specific parts of a JSON object for a given tenant.
```sql theme={null}
UPDATE products
SET details = jsonb_set(details, '{description}', '"A fast and powerful smartphone with a great processor"'::jsonb)
WHERE tenant_id = '550e8400-e29b-41d4-a716-446655440000' AND name = 'Smartphone';
```
This updates the price for the product belonging to a specific tenant.
### **5. Deleting a key from a `jsonb` object by tenant**
You can remove keys from a `jsonb` object for a specific tenant:
```sql theme={null}
-- Remove a single key:
UPDATE products
SET details = details - 'price'
WHERE tenant_id = '550e8400-e29b-41d4-a716-446655440000' AND name = 'Smartphone';
-- Remove multiple keys:
UPDATE products
SET details = details #- '{price, model}'
WHERE tenant_id = '550e8400-e29b-41d4-a716-446655440000' AND name = 'Smartphone';
```
This will delete the keys from the JSON object for the specified tenant.
### **6. Indexing `jsonb` data by `tenant_id` for faster queries**
You can create indexes on the `jsonb` fields for better query performance, particularly for multi-tenant systems.
### **GIN Index (Generalized Inverted Index)**
This index allows fast searches for keys and values within `jsonb` fields for tenants:
```sql theme={null}
CREATE INDEX idx_products_details ON products USING GIN (details);
```
You can now query efficiently using operators like `@>`, and PostgreSQL will leverage this index.
### **7. Aggregating data from `jsonb` by tenant**
You can extract and aggregate data from `jsonb` fields for each tenant.
```sql theme={null}
SELECT tenant_id, details->>'brand', COUNT(*)
FROM products
GROUP BY tenant_id, details->>'brand';
```
This query counts the number of products per brand for each tenant.
### **8. Full-Text Search on `jsonb` fields by tenant**
You can perform full-text searches on `jsonb` columns for specific tenants using `to_tsvector()`.
```sql theme={null}
SELECT * FROM products
WHERE tenant_id = '550e8400-e29b-41d4-a716-446655440000'
AND to_tsvector(details->>'description') @@ to_tsquery('fast & processor');
```
This will search for products with the words "fast" and "processor" in the `description` key for a specific tenant.
***
### **Common `jsonb` functions & operators with tenant ID:**
* **`>`**: Accesses a key, returns JSON data.
* **`>>`**: Accesses a key, returns text.
* **`@>`**: Checks containment (whether a JSON object contains another).
* **`jsonb_set()`**: Updates a value for a key in a JSON object.
* **\`\`**: Deletes a key from a JSON object.
* **`||`**: Concatenates two JSON objects.
* **`jsonb_array_elements()`**: Expands a JSON array to a set of rows.
***
### **Best Practices for `jsonb` with Multi-Tenancy:**
1. **Indexing**: Use GIN or BTREE indexes for frequent lookups by `tenant_id` and specific keys.
2. **Normalization**: Use `jsonb` for semi-structured data. For structured data, use regular columns and normalize the schema.
3. **Use UUID for `tenant_id`**: Ensure the `tenant_id` is a UUID to uniquely identify tenants, providing clear data isolation.
# Numeric types in Postgres
Source: https://thenile.dev/docs/postgres/datatype/numeric
## 1. Integer-Point Types
Integer types are used to store whole numbers (integers) without any decimal points. PostgreSQL provides several types of integers with different ranges and storage requirements.
### Subtypes and Examples:
* **`SMALLINT`**: A 2-byte integer.
* **Range**: -32,768 to 32,767
* **Example**:
```sql theme={null}
CREATE TABLE example_smallint (
id SERIAL PRIMARY KEY,
small_value SMALLINT
);
INSERT INTO example_smallint (small_value) VALUES (32767), (-32768);
```
* **`INTEGER` (or `INT`)**: A 4-byte integer.
* **Range**: -2,147,483,648 to 2,147,483,647
* **Example**:
```sql theme={null}
CREATE TABLE example_int (
id SERIAL PRIMARY KEY,
age INTEGER
);
INSERT INTO example_int (age) VALUES (25), (2147483647), (-2147483648);
```
* **`BIGINT`**: An 8-byte integer.
* **Range**: -9,223,372,036,854,775,808 to 9,223,372,036,854,775,807
* **Example**:
```sql theme={null}
CREATE TABLE example_bigint (
id BIGSERIAL PRIMARY KEY,
large_value BIGINT
);
INSERT INTO example_bigint (large_value) VALUES (9223372036854775807), (-9223372036854775808);
```
* **`SERIAL`, `BIGSERIAL`, `SMALLSERIAL`**: Auto-incrementing integer types.
* **`SERIAL`**: 4-byte integer with auto-increment.
```sql theme={null}
CREATE TABLE example_serial (
id SERIAL PRIMARY KEY
);
```
* **`BIGSERIAL`**: 8-byte integer with auto-increment.
```sql theme={null}
CREATE TABLE example_bigserial (
id BIGSERIAL PRIMARY KEY
);
```
### Operations:
* **Arithmetic Operations**:
```sql theme={null}
SELECT age + 5 FROM example_int;
SELECT large_value * 2 FROM example_bigint;
```
* **Comparison Operations**:
```sql theme={null}
SELECT * FROM example_int WHERE age > 30;
SELECT * FROM example_smallint WHERE small_value = 100;
```
### Performance Considerations:
* **Storage Size**: `SMALLINT` (2 bytes), `INTEGER` (4 bytes), `BIGINT` (8 bytes). Choosing the right type can save storage space.
* **Range**: Ensure the type you choose covers the range of values you expect to store. Using `SMALLINT` for large values will result in an overflow error.
* **Auto-increment**: `SERIAL` types are convenient for primary keys, but be aware of the maximum limits for each type (`SERIAL` for `INTEGER`, `BIGSERIAL` for `BIGINT`).
### When to Use:
* **`SMALLINT`**: For small-range integer values to save space (e.g., age, count).
* **`INTEGER`**: For general-purpose whole numbers (e.g., IDs, quantities).
* **`BIGINT`**: For large-range integer values (e.g., financial transactions, large datasets).
* **`SERIAL`**: For auto-incrementing primary keys with `INTEGER` range.
* **`BIGSERIAL`**: For auto-incrementing primary keys with `BIGINT` range.
## 2. Floating-Point Types
Floating-point types are used to store real numbers, which include fractions (decimal points). PostgreSQL provides single and double precision floating-point numbers.
### Subtypes and Examples:
* **`REAL`**: A 4-byte single-precision floating-point number.
* **Range**: Approximately ±3.40282347E+38 (7 decimal digits precision)
* **Example**:
```sql theme={null}
CREATE TABLE example_real (
id SERIAL PRIMARY KEY,
height REAL
);
INSERT INTO example_real (height) VALUES (1.75), (3.14);
```
* **`DOUBLE PRECISION`**: An 8-byte double-precision floating-point number.
* **Range**: Approximately ±1.7976931348623157E+308 (15 decimal digits precision)
* **Example**:
```sql theme={null}
CREATE TABLE example_double (
id SERIAL PRIMARY KEY,
weight DOUBLE PRECISION
);
INSERT INTO example_double (weight) VALUES (70.5), (150.75);
```
### Operations:
* **Arithmetic Operations**:
```sql theme={null}
SELECT height * 2 FROM example_real;
SELECT weight / 2 FROM example_double;
```
* **Comparison Operations**:
```sql theme={null}
SELECT * FROM example_double WHERE weight > 70.5;
SELECT * FROM example_real WHERE height = 1.75;
```
### Performance Considerations:
* **Precision**: `REAL` has less precision than `DOUBLE PRECISION`. Use `DOUBLE PRECISION` for calculations requiring high precision.
* **Storage Size**: `REAL` (4 bytes), `DOUBLE PRECISION` (8 bytes).
* **Approximation**: Floating-point numbers can introduce rounding errors. For exact values, consider using `NUMERIC`.
### When to Use:
* **`REAL`**: For approximate values where precision is not critical and storage space is a concern (e.g., scientific measurements).
* **`DOUBLE PRECISION`**: For scientific calculations requiring higher precision (e.g., financial models, simulations).
## 3. Exact Numeric Types
Exact numeric types are used to store numbers with a fixed number of decimal places, making them suitable for financial and monetary data.
### Subtypes and Examples:
* **`NUMERIC` (or `DECIMAL`)**: Stores exact numbers with an arbitrary precision.
* **Range**: Specified by `NUMERIC(p, s)` where `p` is the total number of digits and `s` is the number of digits to the right of the decimal point.
* **Example**:
```sql theme={null}
CREATE TABLE example_numeric (
id SERIAL PRIMARY KEY,
price NUMERIC(10, 2)
);
INSERT INTO example_numeric (price) VALUES (12345.67), (99999.99);
```
### Operations:
* **Arithmetic Operations**:
```sql theme={null}
SELECT price * 1.1 FROM example_numeric;
SELECT price + 100 FROM example_numeric;
```
* **Comparison Operations**:
```sql theme={null}
SELECT * FROM example_numeric WHERE price < 100.00;
SELECT * FROM example_numeric WHERE price = 12345.67;
```
### Performance Considerations:
* **Precision and Scale**: `NUMERIC(p, s)` allows you to define the precision (`p` total digits) and scale (`s` digits after the decimal point). This makes it suitable for financial calculations where exact values are crucial.
* **Storage Size**: Storage size varies based on the precision and scale defined. Generally, the more precise the number, the more storage it will require.
### When to Use:
* **`NUMERIC`**: For financial and monetary data requiring exact precision (e.g., currency values, financial calculations).
***
## Summary
Here’s a quick summary of the PostgreSQL numeric types covered:
* **Integer Types**:
* **`SMALLINT`**: Small-range integers, 2 bytes.
* **`INTEGER`**: General-purpose integers, 4 bytes.
* **`BIGINT`**: Large-range integers, 8 bytes.
* **`SERIAL`**: Auto-incrementing integers, 4 bytes.
* **`BIGSERIAL`**: Auto-incrementing large integers, 8 bytes.
* **Floating-Point Types**:
* **`REAL`**: Single-precision floating-point, 4 bytes.
* **`DOUBLE PRECISION`**: Double-precision floating-point, 8 bytes.
* **Exact Numeric Types**:
* **`NUMERIC` (or `DECIMAL`)**: Exact numbers with arbitrary precision.
By understanding and using these numeric types appropriately, you can effectively manage and store numeric data in your PostgreSQL database. This ensures that your database is both efficient and reliable.
# Indexes in Postgres
Source: https://thenile.dev/docs/postgres/indexes
Indexes in PostgreSQL enhance database performance by allowing faster retrieval of specific rows. They work like an index in a book, providing quick references to relevant data. Here are the main index types:
## B-Tree Index
B-Tree indexes play a crucial role in enhancing database performance by allowing faster retrieval of specific rows. Imagine them as the index pages in a book, providing quick references to relevant data.
**1. Structure of B-Tree Indexes:**
* B-tree indexes are organized as balanced tree structures.
* Each level of the tree acts like a doubly-linked list of pages.
* The index starts with a metapage at the beginning of the first segment file.
* All other pages are either leaf pages (the lowest level) or internal pages.
**2. Behavior and Use Cases:**
* B-trees are versatile and widely applicable:
* **Equality and Range Queries**: They excel in handling equality and range queries. Common operators include `=`, `<`, `>`, `BETWEEN`, and `IN`.
* **NULL Conditions**: B-trees can handle `IS NULL` or `IS NOT NULL` conditions.
* **Pattern Matching**: When anchored to the beginning of a string, they efficiently support pattern matching using `LIKE` or `~`.
**3. Practical Examples:**
Let's create an example `employees` table and demonstrate B-tree index usage:
```sql theme={null}
CREATE TABLE employees (
emp_id SERIAL PRIMARY KEY,
emp_name VARCHAR(255) NOT NULL,
emp_salary NUMERIC(10, 2) NOT NULL
);
-- Insert some data
INSERT INTO employees (emp_name, emp_salary) VALUES
('Alice', 60000.00),
('Bob', 75000.00),
('Charlie', 90000.00);
-- Create an index on emp_name
CREATE INDEX employees_name ON employees(emp_name);
-- Query using the index
SELECT * FROM employees WHERE emp_name = 'Bob';
```
The output will be:
```
emp_id | emp_name | emp_salary
--------+----------+------------
2 | Bob | 75000.00
```
## Hash Index
Hash indexes use a hash function to map indexed column values to 32-bit hash codes. These indexes are optimized for simple equality comparisons (using the `=` operator). Here's how they work:
1. **Structure**:
* Hash indexes store only the hash value of the data being indexed.
* No restrictions on the size of the indexed column.
* Support only single-column indexes.
* Do not allow uniqueness checking.
2. **Use Cases**:
* Ideal for scenarios where exact matches are common.
* Not suitable for range queries or pattern matching.
3. **Performance Considerations**:
* Fast for equality lookups.
* Minimal overhead during data insertion.
* Not automatically maintained (unlike B-tree indexes).
## Example: Employees Table
Let's create an example `employees` table and demonstrate Hash index usage:
```sql theme={null}
CREATE TABLE employees (
emp_id SERIAL PRIMARY KEY,
emp_name VARCHAR(255) NOT NULL,
emp_salary NUMERIC(10, 2) NOT NULL
);
-- Insert some data
INSERT INTO employees (emp_name, emp_salary) VALUES
('Alice', 60000.00),
('Bob', 75000.00),
('Charlie', 90000.00);
-- Create a Hash index on emp_name
CREATE INDEX employees_name_hash ON employees USING HASH (emp_name);
-- Query using the index
SELECT * FROM employees WHERE emp_name = 'Bob';
```
Output:
```
emp_id | emp_name | emp_salary
--------+----------+------------
2 | Bob | 75000.00
```
## GiST Index
GiST indexes are a versatile type of index that can handle complex data types, such as geometric shapes, full-text search, and network addresses. [They are implemented using a custom data structure optimized for searching large amounts of data1](https://www.postgresql.org/docs/current/gist.html). Here are some key points:
* **Purpose**: GiST indexes are designed to support various query types, including equality queries, range queries, and partial match queries.
* **Infrastructure**: GiST provides an infrastructure within which different indexing strategies can be implemented.
* [**Operator Classes**: The operators used with GiST indexes depend on the specific indexing strategy (operator class) chosen2](https://www.postgresql.org/docs/current/indexes-types.html).
## Examples of GiST Indexes
Let's explore some examples using tables related to employees. We'll create a sample table, insert data, and demonstrate how GiST indexes work.
### Example 1: Geometric Shapes
Suppose we have an `employees` table with a `location` column representing the employees' office locations (stored as geometric points). We want to efficiently query employees based on their proximity to a specific location.
1. **Table Creation**:
```sql theme={null}
CREATE TABLE employees (
id SERIAL PRIMARY KEY,
name VARCHAR(100),
location POINT
);
```
2. **Insert Data**:
```sql theme={null}
INSERT INTO employees (name, location)
VALUES
('Alice', POINT(1, 2)),
('Bob', POINT(3, 4)),
-- ... (more data)
;
```
3. **Create GiST Index**:
```sql theme={null}
CREATE INDEX idx_location ON employees USING GIST (location);
```
4. **Query Using Index**:
```sql theme={null}
SELECT name
FROM employees
WHERE location <-> POINT(2, 3) < 1;
```
This query retrieves employees within 1 unit of distance from the point (2, 3).
## Performance Considerations
* GiST indexes are powerful but may have higher insertion and maintenance costs compared to B-tree indexes.
* Choose the appropriate operator class and indexing strategy based on your data type and query requirements.
## SP-GiST
SP-GiST (Spatial Generalized Search Tree) indexes are a versatile index type offered by PostgreSQL. They are designed for complex, non-rectangular data types and work especially well with geometrical and network-based data. Here are some key points:
1. **Infrastructure**: SP-GiST indexes support various kinds of searches, similar to GiST indexes. [They permit the implementation of a wide range of different non-balanced disk-based data structures, such as quadtrees, k-d trees, and radix trees (tries) 1](https://www.postgresql.org/docs/current/spgist.html).
2. **Use Cases**:
* **Geometric Searches**: SP-GiST is ideal for spatial data, such as points, lines, and polygons.
* **IP Network Searches**: When dealing with IP addresses or network ranges.
* [**Text Search with Complex Pattern Matching**: For scenarios where you need to search for patterns within text data](https://www.postgresql.org/docs/current/spgist.html) [2](https://roadmap.sh/postgresql-dba/sql-optimization-techniques/indexes-usecases/sp-gist).
3. **Performance Considerations**:
* SP-GiST indexes are most useful for data that has a natural clustering element and is not an equally balanced tree.
* [They work well with data types that don't fit neatly into rectangular shapes, like GIS (geospatial), multimedia, phone routing, and IP routing data 3](https://www.postgresqltutorial.com/postgresql-indexes/postgresql-index-types/).
## Example: Employee Table
Let's create an example employee table and demonstrate how to use SP-GiST indexes.
### 1. Create the Employee Table
```sql theme={null}
CREATE TABLE employees (
emp_id SERIAL PRIMARY KEY,
emp_name VARCHAR(100),
emp_location POINT
);
```
### 2. Insert Sample Data
```sql theme={null}
INSERT INTO employees (emp_name, emp_location)
VALUES
('Alice', POINT(10, 20)),
('Bob', POINT(15, 25)),
('Charlie', POINT(30, 40));
```
### 3. Create an SP-GiST Index on `emp_location`
```sql theme={null}
CREATE INDEX idx_emp_location ON employees USING spgist(emp_location);
```
### 4. Query Using the Index
### Find Employees Near a Given Point
```sql theme={null}
SELECT emp_name
FROM employees
WHERE emp_location <-> POINT(12, 22) < 5;
```
This query finds employees whose location is within 5 units of the point (12, 22).
### Output:
emp\_name
***
Alice
***
Bob
***
### 5. Another Example: IP Network Search
Suppose we have an IP address range column:
```sql theme={null}
CREATE TABLE network_devices (
device_id SERIAL PRIMARY KEY,
device_name VARCHAR(100),
ip_range inet
);
INSERT INTO network_devices (device_name, ip_range)
VALUES
('Router A', '192.168.1.0/24'),
('Switch B', '10.0.0.0/16'),
('Firewall C', '172.16.0.0/20');
CREATE INDEX idx_ip_range ON network_devices USING spgist (ip_range);
```
```sql theme={null}
SELECT device_name
FROM network_devices
WHERE ip_range >> '192.168.1.42';
```
This query finds devices whose IP range includes the address '192.168.1.42'.
### Output:
device\_name
***
Router A
## GIN Index
A GIN index is designed for efficiently handling composite data values, such as arrays or JSON objects. Here are the key points:
1. **What is a GIN Index?**
* A GIN index stores a set of `(key, posting list)` pairs.
* The posting list contains row IDs where the key occurs.
* Multiple posting lists can share the same row ID since an item can have multiple keys.
* [Each key value is stored only once, making GIN indexes compact when the same key appears multiple times](https://www.postgresql.org/docs/current/gin-intro.html).
2. **Use Cases for GIN Indexes:**
* GIN indexes are ideal for data values with multiple components, like arrays.
* [They efficiently handle queries that search for specific component values within composite items](https://www.postgresql.org/docs/current/gin-intro.html).
3. **Example 1: Basic Query Using GIN Index**
* Suppose we have an `employees` table with a column `skills` (an array of skills). Let's create an `employees` table:
```sql theme={null}
CREATE TABLE employees (
id SERIAL PRIMARY KEY,
name VARCHAR(100),
skills TEXT[] -- Array of skills
);
```
* Insert some data:
```sql theme={null}
INSERT INTO employees (name, skills)
VALUES
('Alice', ARRAY['Java', 'SQL']),
('Bob', ARRAY['Python', 'JavaScript']);
```
* To create a GIN index on the `skills` column:
```sql theme={null}
CREATE INDEX idx_gin_skills ON employees USING gin(skills);
```
* Query using the GIN index:
```sql theme={null}
SELECT name FROM employees WHERE skills @> ARRAY['Java'];
```
* Output: `Alice`
1. **Example 2: Searching for Multiple Skills**
* Query to find employees with both Java and SQL skills:
```sql theme={null}
SELECT name FROM employees WHERE skills @> ARRAY['Java', 'SQL'];
```
* Output: `Alice`
2. **Example 3: Partial Match**
* Query to find employees with any of the specified skills:
```sql theme={null}
SELECT name FROM employees WHERE skills && ARRAY['Python', 'JavaScript'];
```
* Output: `Bob`
3. **Performance Considerations:**
* GIN indexes are efficient for array-based queries but may have overhead during updates.
* Consider the trade-off between query performance and update cost.
* Regularly vacuum the GIN index to maintain performance.
## BRIN Index
* **BRIN** stands for **Block Range Index**.
* Designed for handling very large tables with columns that have natural correlation to their physical location within the table.
* Works in terms of **block ranges** (or "page ranges").
* Each block range groups physically adjacent pages in the table.
* Summary information is stored by the index for each block range.
* **Lossy**: BRIN indexes can satisfy queries via regular bitmap index scans but are lossy, meaning the query executor rechecks tuples and discards those not matching query conditions.
* Size of block range determined at index creation time by `pages_per_range` storage parameter.
## Use Cases for BRIN Indexes
1. **Time-Series Data**: Ideal for tables with a timestamp column (e.g., sales orders, logs).
2. **Geospatial Data**: Useful for tables with spatial data (e.g., ZIP codes, geographical coordinates).
## Example: Employee Table
Let's create an employee table and demonstrate BRIN index usage.
### 1. Create Employee Table
```sql theme={null}
CREATE TABLE employees (
emp_id SERIAL PRIMARY KEY,
emp_name VARCHAR(100),
hire_date DATE
);
```
### 2. Insert Sample Data
```sql theme={null}
INSERT INTO employees (emp_name, hire_date)
VALUES
('Alice', '2022-01-15'),
('Bob', '2021-03-10'),
('Charlie', '2020-11-20');
```
### 3. Create BRIN Index on `hire_date`
```sql theme={null}
CREATE INDEX idx_employees_hire_date_brin
ON employees USING brin (hire_date);
```
### 4. Query Using BRIN Index
```sql theme={null}
-- Find employees hired after 2021-01-01
SELECT emp_name
FROM employees
WHERE hire_date >= '2021-01-01';
```
### Output
```
emp_name
----------
Alice
Bob
(2 rows)
```
Remember that BRIN indexes are most effective when dealing with large tables and specific column types.
[For more details, refer to the official PostgreSQL documentation.](https://www.postgresql.org/docs/current/indexes.html)
# Introduction to Postgres
Source: https://thenile.dev/docs/postgres/introduction
This section of Nile documentation is intended to provide a basic introduction to Postgres and common relational database concepts.
If you are new to relational databases or to Postgres, this is a great place to start. You'll learn the basic data structures, operations, tools,\
and best practices for working with Postgres.
In this section, we will cover the following topics:
* [**Creating and Altering Tables**](postgres/createtable): Tables are the basic building block of a relational database. Tables (sometimes referred to as relations or tuples) are used to store data in rows and columns.
We will cover how to create tables, add columns and define constraints such as primary keys and foreign keys.
* [**Data Types**](postgres/datatype/): Postgres supports a wide range of data types for storing different types of data. In a table, each column has a data type, and based on this data type, Postgres allocates storage and allows various operations.
We will cover some of the most common data types and how to use them.
* [**Indexes**](postgres/indexes): Indexes are data structures that improve the speed of data retrieval operations on a table. They help to optimize queries by
allowing the database to find rows more quickly. We will cover how to create indexes and when to use them.
* [**Joins**](postgres/joins): Joins are used to combine rows from two or more tables based on a related column between them.
They are essential for querying data from multiple tables. We will cover different types of joins and how to use them.
* [**Views**](postgres/views): Views are virtual tables that are generated based on the result of a query. They provide a way to present data from one or more tables in a structured format.
We will cover how to create views and use them in your database.
* [**Ecosystem tools**](postgres/tools): Postgres has a rich ecosystem of tools that make it easy to work with. We will cover some popular tools that can help you connect to Postgres and work with Postgres more effectively.
# Joins in Postgres
Source: https://thenile.dev/docs/postgres/joins
### What Are Joins?
Joins in PostgreSQL allow you to combine data from multiple tables based on matching values in specific columns. They provide a way to query data across related tables. To explain the join concepts, let us take the example of employees table and departments. We will use these two tables to explain different Join types.
### Tables: `employees` and `departments`
```sql theme={null}
CREATE TABLE employees (
employee_id INT PRIMARY KEY,
employee_name VARCHAR(100) NOT NULL,
department_id INT
);
CREATE TABLE departments (
department_id INT PRIMARY KEY,
department_name VARCHAR(100) NOT NULL
);
INSERT INTO employees (employee_id, employee_name, department_id)
VALUES (1, 'Alice', 101), (2, 'Bob', 102), (3, 'Carol', NULL);
INSERT INTO departments (department_id, department_name)
VALUES (101, 'HR'), (102, 'Engineering'), (103, 'Finance');
```
### Common Types of Joins:
1. **Inner Join**:
* Returns records with matching values in both tables.
* Example: Combining `employees` and `departments` tables based on matching department IDs.
* SQL:
```sql theme={null}
SELECT e.employee_id, e.employee_name, d.department_name
FROM employees e
INNER JOIN departments d ON e.department_id = d.department_id;
```
* Output:
```
employee_id | employee_name | department_name
------------+---------------+-----------------
1 | Alice | HR
2 | Bob | Engineering
```
2. **Left Join (Left Outer Join)**:
* Includes all records from the left table and matched records from the right table.
* If no match exists, NULL values are returned for right table columns.
* Example: Retrieving all employees along with their department names (even if they don't belong to any department).
* SQL:
```sql theme={null}
SELECT e.employee_id, e.employee_name, d.department_name
FROM employees e
LEFT JOIN departments d ON e.department_id = d.department_id;
```
* Output:
```
employee_id | employee_name | department_name
------------+---------------+-----------------
1 | Alice | HR
2 | Bob | Engineering
3 | Carol | NULL
```
3. **Right Join (Right Outer Join)**:
* Similar to left join but prioritizes the right table.
* Example: Retrieving all departments along with their employees (even if there are no employees in a department).
* SQL:
```sql theme={null}
SELECT e.employee_id, e.employee_name, d.department_name
FROM employees e
RIGHT JOIN departments d ON e.department_id = d.department_id;
```
* Output:
```
employee_id | employee_name | department_name
------------+---------------+-----------------
1 | Alice | HR
2 | Bob | Engineering
NULL | NULL | Finance
```
4. **Full Outer Join**:
* Combines results from both left and right outer joins.
* Includes unmatched records from both tables.
* Example: Getting all employees and their department names.
* SQL:
```sql theme={null}
SELECT e.employee_id, e.employee_name, d.department_name
FROM employees e
FULL OUTER JOIN departments d ON e.department_id = d.department_id;
```
* Output:
```
employee_id | employee_name | department_name
------------+---------------+-----------------
1 | Alice | HR
2 | Bob | Engineering
3 | Carol | NULL
NULL | NULL | Finance
```
### Performance Considerations:
* Use smaller data types to reduce memory and disk space usage.
* Tune PostgreSQL settings (e.g., shared\_buffers, work\_mem) for caching and sorting.
* [Consider using faster storage (e.g., SSDs) for disk-intensive joins](https://www.cybertec-postgresql.com/en/join-strategies-and-performance-in-postgresql/).
# Postgres Compatibility
Source: https://thenile.dev/docs/postgres/postgres-compatibility
# Overview
Nile's Postgres is built 100% on Postgres and is protocol and application-compatible with Postgres. However, given it is a serverless database, there are some restrictions that you should be aware of. Nile currently supports Postgres 15.
# Current limitations
The table below lists all the features in Postgres that are not yet supported in Nile's Postgres. We are continuing to work on features focused on SaaS use cases. If you have any feature need, we would love to hear from you. Please add your upvote to existing feature requests or add your own on our [Github discussion](https://github.com/orgs/niledatabase/discussions/categories/feature-requests) forum.
Statement
Description
What Nile Recommends
CREATE FUNCTION
User defined functions are not supported yet
Push the logic to the application
CREATE TRIGGER
Triggers are not supported yet since UDF support is not tdere
We hope to support real time events soon
CREATE SEQUENCE
Not supported for tenant tables. Supported for shared tables
CREATE POLICY
Nile will have the ability to define policies through a more powerful
permission system that is in progress
Would love to hear your use case for this in our github discussion forum
CREATE ROLE
Ability to define granular roles for developers using the database
Developer roles can be supported in Nile's console when we support it
CREATE USER
Ability to create a new developer for a database
Creating a new developer to use the database is supported in Nile's
console (and API/CLI in the future)
CREATE DATABASE
Ability to create a new databasee
You can do this from Nile's console
Transactional writes between tenant and shared tables
Writes across a tenant and shared in the same transaction
this is not common from the users we have spoken to. Usually shared
tables are loaded separately from an external source. Would love to hear
your use case for this in our github discussion forum
Foreign keys from tenant table referencing shared table
Referencing a column from a tenant table to a shared table
Would love to hear your use case for this in our github discussion forum
Transactional writes across different tenants
Writes across two or more tenants in the same transaction
this is not common from the users we have spoken to. Most usecases have
transactions with the same tenant. Would love to hear your use case for
this in our github discussion forum
# Additional help
If you need additional help or have any questions, join the Nile community [Discord](https://discord.gg/8UuBB84tTy) or [Github discussion](https://github.com/orgs/niledatabase/discussions) forum, or send mail to [support@thenile.dev](mailto:support@thenile.dev).
# Connecting to Postgres using DBeaver
Source: https://thenile.dev/docs/postgres/tools/dbeaver
This guide will walk you through the steps to connect to Nile using PSQL.
1. Create a Nile's Postgres database through the UI or use one of your existing databases to follow along.
2. Once you are inside a specific database, click on the settings tab on the left and navigate to the connections screen
3. Click on the Postgres icon to get the connection string
4. Click "Generate credentials". Use this if you really want to simplify the access. If not, you can replace the username and password manually and use that through commandline.
5. There are four parts of this url that is needed to connect to DBWeaver
```sql theme={null}
psql postgres://018bd797-ed9d-7c02-bec4-dac1179c29fd:d68af75e-47cb-4354-ada4-98abe7d8015c@db.thenile.dev:5432/diplomatic_axe
host:db.thenile.dev
database:diplomatic_axe
username:018bd797-ed9d-7c02-bec4-dac1179c29fd
password:d68af75e-47cb-4354-ada4-98abe7d8015c
```
6. Navigate to DBWeaver. Select to connect to a new database
7. Pick Postgres as the choice of database
8. Enter the host, database, username and password from step 5
9. Press OK and DBWeaver should automatically connect to Nile's Postgres
10. Enjoy exploring Nile with DBWeaver
# Connecting to Postgres using PgAdmin
Source: https://thenile.dev/docs/postgres/tools/pgadmin
This guide will walk you through the steps to connect to Nile using PgAdmin.
1. Create a Nile's Postgres database through the UI or use one of your existing databases to follow along.
2. Once you are inside a specific database, click on the settings tab on the left and navigate to the connections screen
3. Click on the Postgres icon to get the connection string
4. Click "Generate credentials". Use this if you really want to simplify the access. If not, you can replace the username and password manually and use that through commandline.
5. There are four parts of this url that is needed to connect to PgAdmin
```bash theme={null}
host:db.thenile.dev
database:diplomatic_axe
username:018bd797-ed9d-7c02-bec4-dac1179c29fd
password:d68af75e-47cb-4354-ada4-98abe7d8015c
```
6. Navigate to PgAdmin. Select to create a new server. Provide a name for the server
7. Enter the host, database, username and password from step 5
8. Press OK and PgAdmin should automatically connect to Nile's Postgres
10. Known issues
* You cannot create a table through the UI editor. You can create a table through the query editor
* You cannot get information about indexes or constraints through the UI editor
11. Enjoy exploring Nile with PgAdmin
# Connecting to Postgres using PSQL
Source: https://thenile.dev/docs/postgres/tools/psql
This guide will walk you through the steps to connect to Nile using PSQL.
1. Create a Nile's Postgres database through the UI or use one of your existing databases to follow along.
2. Once you are inside a specific database, click on the settings tab on the left and navigate to the connections screen
3. Click on the Postgres icon to get the connection string
4. Click "Generate credentials". Use this if you really want to simplify the access. If not, you can replace the username and password manually and use that through commandline.
5. Copy the connection string and run it on the command line.
6. You can now use the full power of psql with Nile. You can take a look at all the tables and schemas using \dt
7. Execute any queries to interact with the tables. For example, if you want to see the rows in tenants table, execute a select query on it
8. Have fun using PSQL!
# Views in Postgres
Source: https://thenile.dev/docs/postgres/views
Views in PostgreSQL are virtual tables that are generated based on the result of a query.
They provide a way to present data from one or more tables in a structured format. Views are useful for simplifying complex queries,
providing an additional layer of security, and encapsulating complex logic.
### Creating a View
To create a view in PostgreSQL, you use the `CREATE VIEW` statement followed by the view name and the query that defines the view.
For example, assuming you have two tables `employees` and `departments` (we created them in the [tables section](./createtable) of this guide),
you can create a view that combines data from both tables:
```sql theme={null}
CREATE VIEW employee_details AS
SELECT employees.employee_id,
employees.first_name,
employees.last_name,
departments.department_name
FROM employees
JOIN departments ON employees.department_id = departments.department_id;
```
This view provides an abstraction over the underlying tables and allows you to query the combined data without having to write the full join
every time. You can query the view just like a regular table:
```sql theme={null}
SELECT * FROM employee_details;
```
### Updating a View
There are two ways to update a view in PostgreSQL:
#### Recreating the View
You can update a view by replacing it with the updated query. This allows you to modify the underlying query, as long as the column
names and data types of existing columns remain the same and in the same order. You can add new columns to the end of the `SELECT`
statement, and also modify the underlying logic behind the columns as long as the name and type remain the same.
Because the view does not contain any data itself, replacing it is a fast, atomic and idempotent (repeatable) operation.
Because the limitations prevent you from making any breaking changes, the operation is safe to perform online (while the application is running).
For example, if you want to include the employee email in the `employee_details` view, you can replace the view as follows:
```sql theme={null}
CREATE OR REPLACE VIEW employee_details AS
SELECT employees.employee_id,
employees.first_name,
employees.last_name,
departments.department_name,
employees.email
FROM employees
JOIN departments ON employees.department_id = departments.department_id;
```
The email column is added to the view definition, and the view is replaced with the updated query. We have to add the new column as the
last column in the SELECT statement to avoid breaking changes.
#### Altering a View
Some properties of a view can be altered without replacing the view definitions, for example you can change a view name, owner,
rename a column or modify default values.
For example:
```sql theme={null}
ALTER VIEW employee_details RENAME COLUMN department_name TO dept_name;
```
### Dropping a View
To drop a view in PostgreSQL, you use the `DROP VIEW` statement followed by the view name:
```sql theme={null}
DROP VIEW employee_details;
```
Dropping a view and re-creating it is also the way to modify a view in ways that are not supported by the `CREATE OR REPLACE VIEW` syntax.
You can do this in a single transaction to ensure that the view is always available for querying, but be aware that if you remove columns
that are still in use, you may break the application.
```sql theme={null}
BEGIN;
DROP VIEW employee_details;
-- note that the below view definition has different columns from the original view
CREATE VIEW employee_details AS
SELECT employees.employee_id,
employees.first_name || ' ' || employees.last_name as full_name,
employees.email,
departments.department_name
FROM employees
JOIN departments ON employees.department_id = departments.department_id;
COMMIT;
```
# Backup and Restore
Source: https://thenile.dev/docs/support/backup_restore
In case database backup and restore is required, please follow the steps below. Our team is here to assist you throughout the process.
This is available for paid tiers only. We hope to make this process self-serve
soon.
## 1. Notify Us
Send a message to our team via Slack (if we have set up one for you) or email at [support@thenile.dev](mailto:support@thenile.dev) and include **URGENT** in the title. Include the following details to expedite the recovery process:
* Database name
* Approximate time when the issue occurred
* Do you wish to restore to new schema in same database or to a different database? (if a different database, please provide its name as well).
## 2. Acknowledgment
Once we receive your message, we will acknowledge it within **15 minutes**. If you do not receive an acknowledgment, please follow up.
## 3. Backup Availability
We take **three backups daily** to ensure your data is well-protected. Upon your request, an appropriate backup, based on the time when the data issue occurred, will be restored in a separate schema within your database or alternatively into the original schema in an empty database of your choice.
In the case of separate schema, the backup schema will have a name in the format: `_bkp`.
For example, if your original schema is `sales_data`, the backup schema will be named: `sales_data_bkp`
## 4. How to Restore Data
Only needed if data is restored within the same database into another schema
You can restore the data from the backup schema using SQL commands. Below is an example query to copy data back to the original schema:
```sql theme={null}
-- Recover data for a specific tenant:
SET nile.tenant_id='018ff561-ae16-7c9e-93c6-8482f9952c4f';
INSERT INTO expense_report_items
SELECT * FROM public_bkp.expense_report_items;
-- recover data for shared table
RESET nile.tenant_id;
INSERT INTO flights select * from flights;
-- recover data for multiple tenants:
-- 1. generate statements:
RESET nile.tenant_id;
SELECT 'SET nile.tenant_id='''||id||'''; INSERT INTO expense_report_items SELECT * FROM public_bkp.expense_report_items;' from tenants where name like '%customer%';
-- 2. Run the resulting statements:
SET nile.tenant_id='179027f2-e184-4df7-a568-2be746898be2'; INSERT INTO expense_report_items SELECT * FROM public_bkp.expense_report_items;
SET nile.tenant_id='018ff04c-9dfe-7fbd-8d1c-96ccdbe24e11'; INSERT INTO expense_report_items SELECT * FROM public_bkp.expense_report_items;
SET nile.tenant_id='018ff04d-7ded-7719-a2d3-79f65d19f2e8'; INSERT INTO expense_report_items SELECT * FROM public_bkp.expense_report_items;
SET nile.tenant_id='108124a5-2e34-418a-9735-b93082e9fbf2'; INSERT INTO expense_report_items SELECT * FROM public_bkp.expense_report_items;
SET nile.tenant_id='84a4358a-15a4-42ba-8606-ca948fb730c2'; INSERT INTO expense_report_items SELECT * FROM public_bkp.expense_report_items;
SET nile.tenant_id='a02748b7-76c9-4765-b230-dea566e38e5d'; INSERT INTO expense_report_items SELECT * FROM public_bkp.expense_report_items;
SET nile.tenant_id='cea8b9b8-ec0a-4144-ae3a-b67834079951'; INSERT INTO expense_report_items SELECT * FROM public_bkp.expense_report_items;
```
## Need Assistance?
If you encounter any issues during the recovery process or require additional help, please do not hesitate to contact our team. We’re here to help ensure a smooth recovery.
# Support
Source: https://thenile.dev/docs/support/support
We are in technical preview but would love to support and help developers build world-class SaaS applications. We would love to hear feedback about the experience, the features, bugs and any other cool ideas you might have.
If you need help or have any questions, please reach out to us through one of the following options
1. Join the Nile [Discord](https://discord.gg/8UuBB84tTy) community to ask questions and discuss with other developers using Nile.
2. Participate in our [Github discussion](https://github.com/orgs/niledatabase/discussions) forum to request missing features or if you encountered a bug.
3. If you prefer a private conversation, you can send mail to [support@thenile.dev](mailto:support@thenile.dev).
You can also follow us on [Twitter](https://twitter.com/niledatabase) to get updates from us.
# Tenant Virtualization
Source: https://thenile.dev/docs/tenant-virtualization/introduction
## **What is a tenant?**
A tenant is primarily a company, an organization, or a workspace in your product that contains a group of users. A SaaS application provides services to multiple tenants. Tenant is the basic building block of all SaaS applications. Every SaaS feature and experience is built on top of tenants. Nile has built-in tenant virtualization, which makes it easy, reliable, and cost-effective to develop and support SaaS use cases across the globe.
## How Virtualization works?
# Tenant Isolation
Source: https://thenile.dev/docs/tenant-virtualization/tenant-isolation
All SaaS applications provide service to multiple customers or tenants. Isolating each tenant's data from another tenant is so basic yet hard to do. Nile enables tenant data isolation without any complex, buggy code or dealing with messy row-level security policies. With Postgres being tenant-aware in Nile, this becomes straightforward.
## What is tenant data isolation?
A SaaS application serves many tenants. The data for each tenant belongs to the tenant and should not be accessible to other tenants. The users who belong to a specific tenant can access the data they have access to for that tenant. This seems so obvious but yet so hard to achieve in practice.
In Nile, you can think about it as a single Postgres that has many virtual tenant DBs. You can control where these virtual DBs should be placed. This can be on multitenant or dedicated instances. You get to decide while creating the tenant. This can also be in any region of the world. Nile enables isolation with a single line of code in your application. Nile also ensures user authorization is enforced within a tenant if you use our user management (more about this later)
## Enforcing tenant data isolation
Nile makes it really easy to enforce tenant data isolation. Let us take a recruiting SaaS product that helps a company to source, interview and hire candidates for different jobs open to be filled. A simplified version will have a 'candidates' table that tracks all the candidates in flight. The list of candidate is private to each company using the recruiting product. Given that the candidate information needs to be private to each company, let us create a tenant aware table in Nile's Postgres.
```sql theme={null}
create table candidates (
tenant_id uuid,
id integer,
name text,
status text,
resume_url text,
job_applied text,
applied_date TIMESTAMP,
CONSTRAINT FK_tenants FOREIGN KEY(tenant_id) REFERENCES tenants(id),
CONSTRAINT PK_candidates PRIMARY KEY(tenant_id,id));
```
```sql theme={null}
insert into tenants (id, name) values
('018ade1a-7830-7981-b23f-f3a7f7b8f09f','customer1'),
('018ade1a-7843-7e60-9686-714bab650998','customer2'),
(default, 'customer3'),
(default, 'customer4'),
(default, 'customer5');
```
Let us populate the candidate table for couple of tenants. Typically, these candidates will be inserted by the application as candidates apply for the job. We will manually populate them in this case.
```sql theme={null}
-- inserting candidates for tenant 018ade1a-7830-7981-b23f-f3a7f7b8f09f
insert into candidates
(tenant_id,id,name,status, resume_url, job_applied, applied_date)
values
('018ade1a-7830-7981-b23f-f3a7f7b8f09f',1324,'Candidate1','New Application','http://myresume.com/index.html','software engineer','2023-08-22 19:10:25-07'),
('018ade1a-7830-7981-b23f-f3a7f7b8f09f',2356,'Candidate2','Phone Interview','http://myprofile.com/index.html','senior software engineer','2023-07-12 18:09:15-07');
-- inserting candidates for tenant 018ade1a-7843-7e60-9686-714bab650998
insert into candidates
(tenant_id,id,name,status, resume_url, job_applied, applied_date)
values
('018ade1a-7843-7e60-9686-714bab650998',2532,'Candidate1','Onsite','http://checkresume.com/index.html','Principal engineer','2023-07-20 12:08:14-06'),
('018ade1a-7843-7e60-9686-714bab650998',7374,'Candidate2','Final round','http://forinterview.com/index.html','Product manager','2023-06-04 11:08:11-06'),
('018ade1a-7843-7e60-9686-714bab650998',8345,'Candidate3','Recruiter screen','http://career.com/index.html','Saled engineer','2023-09-10 10:08:12-06');
```
Now we have a table of candidates for multiple customers populated.
```sql theme={null}
select * from candidates;
```
| tenant\_id | id | name | status | resume\_url | job\_applied | applied\_date |
| ------------------------------------ | ---- | ----------- | ---------------- | ------------------------------------------------------------------------ | ------------------------ | ------------------------ |
| 018ade1a-7830-7981-b23f-f3a7f7b8f09f | 1324 | Candidate 1 | New Application | [http://myresume.com/index.html](http://myresume.com/index.html) | software engineer | 2023-08-22T19:10:25.000 |
| 018ade1a-7830-7981-b23f-f3a7f7b8f09f | 2356 | Candidate 2 | Phone Interview | [http://myprofile.com/index.html](http://myprofile.com/index.html) | senior software engineer | 2023-07-12T18:09:15.000Z |
| 018ade1a-7843-7e60-9686-714bab650998 | 2532 | Candidate 1 | Onsite | [http://checkresume.com/index.html](http://checkresume.com/index.html) | Principal engineer | 2023-07-20T12:08:14.000Z |
| 018ade1a-7843-7e60-9686-714bab650998 | 7374 | Candidate 2 | Final round | [http://forinterview.com/index.html](http://forinterview.com/index.html) | Product manager | 2023-06-04T11:08:11.000Z |
| 018ade1a-7843-7e60-9686-714bab650998 | 8345 | Candidate 3 | Recruiter screen | [http://career.com/index.html](http://career.com/index.html) | Sales engineer | 2023-09-10T10:08:12.000Z |
In a typical SaaS application, you would want to access the data of a specific tenant when user accesses the application. You also want to make sure the data from another tenant is not seen. This is typically hard without implementing complex permissions at the application level or enforcing row level security policies in Postgres which is hard to maintain, debug and error prone.
Nile has first class support for tenant isolation at for SQL and all our SDKs which makes it very trivial to do. At the SQL layer, setting a session context to a specific tenant is equivalent to pointing the connection to a specific tenant DB. You can still access shared tables in this mode but the other virtual tenant DBs are not accessible. We also provide a very simple user context mode which adds additional security on top of the tenant context. Read more about this in user authorization.
```sql theme={null}
-- set session context to the tenant you want to access.
set nile.tenant_id = '018ade1a-7830-7981-b23f-f3a7f7b8f09f';
select * from candidates;
```
```typescript theme={null}
// get a connection to the virtual tenant DB you want to access.
const tenantNile = nile.withContext({ headers, tenantId });
const candidates = await tenantNile.query('select * from candidates');
```
| tenant\_id | id | name | status | resume\_url | job\_applied | applied\_date |
| ------------------------------------ | ---- | ----------- | --------------- | ------------------------------------------------------------------ | ------------------------ | ------------------------ |
| 018ade1a-7830-7981-b23f-f3a7f7b8f09f | 1324 | Candidate 1 | New Application | [http://myresume.com/index.html](http://myresume.com/index.html) | software engineer | 2023-08-22T19:10:25.000 |
| 018ade1a-7830-7981-b23f-f3a7f7b8f09f | 2356 | Candidate 2 | Phone Interview | [http://myprofile.com/index.html](http://myprofile.com/index.html) | senior software engineer | 2023-07-12T18:09:15.000Z |
## Querying across tenants
While Nile provides tenant isolation, it also provides
the ability for you to query across tenants and shared tables. This is typically
useful during development, production debugging or to do basic analytics across customers.
Our customer dashboard already provides you useful analytics but you can always run
raw SQL queries for more information. When no tenant context is set, the connection
can access across all the tenant DBs irrespective of where the tenants are located
(any region, multitenant or dedicated infrastructure).
1. Querying the total no of customers
```sql theme={null}
-- Calculate the total no of customers signed up for the product
select count(*) as no_of_customers from tenants;
```
| no\_of\_customers |
| ----------------- |
| 5 |
2. Querying the total no of candidates per customer
```sql theme={null}
-- Calculates the total no of candidates per tenant that have
-- applied for a job for a recruiting product. Can be used to define
-- active tenants
select t1.id as customer_id,t1.name as customer_name,
count(c1.id) as no_of_candidates from candidates c1
right join tenants t1 on c1.tenant_id=t1.id group by t1.id,t1.name;
```
| customer\_id | customer\_name | no\_of\_candidates |
| ------------------------------------ | -------------- | ------------------ |
| 018ad859-3977-7c0d-8de1-7c6add417cdb | customer 1 | 2 |
| 018ade1a-7830-7981-b23f-f3a7f7b8f09f | customer 2 | 3 |
| 018ade1a-7843-7e60-9686-714bab650998 | customer 3 | 0 |
3. Finding the number of customers that have atleast one candidate added in June:
```sql theme={null}
select tenant_id from candidates
where extract(month from applied_date)=6;
```
| tenant\_id |
| ------------------------------------ |
| 018ade1a-7843-7e60-9686-714bab650998 |
## What is tenant performance isolation?
Apart from data isolation, another key isolation
for SaaS applications is to ensure the performance of one tenant is not impacting
other tenants. This is usually called noisy neighbor problem. The most trivial way
to achieve this at the database level is to have a separate database per tenant.
There are obvious downsides to this approach from operational overhead (schema rollouts,
monitoring, capacity planning), cost (lower utilization) and application complexity
(metadata management, routing). The other extreme is to have all tenants on the same
database and solve tenant isolation problems by scaling up or application level quotas.
This approach has the benefit of having lesser operational complexity and lower cost.
Practically, all SaaS applications ends up with a hybrid model depending on their
use cases and customer needs.
## How does Nile achieve performance isolation between tenants?
Nile's Postgres by default places tenants in a multitenant model but provides
the flexibility to also place some of the tenants in dedicated infrastructure (coming
soon). The most ideal scenario is to receive predictable workload throughout the
day from all tenants. You could then scale the infrastructure for that workload.
However, in practice, you would typically get random peak workloads from different
tenants. This makes it hard to plan for.
There are different approaches to tackle this problem. You could either over provision for the worst case and waste resources or have the ability to instantly auto scale. Another thing with multitenant workloads is that as demand increases, the total workload tends to flatten out and becomes much more predictable to plan for. As scale increases, this gets easier. Another approach is to move tenants between machines to rebalance depending on workload distribution and infrastructure utilization. Nile employs all these approaches to ensure tenants never get impacted by the performance of other tenants. A future plan is to also allow users to specify the resource quota for each tenant which Nile can enforce. This would help to avoid spending too much resources for some of your tenants (example, your free tier customers can have more aggressive quotas).
# Tenant Management
Source: https://thenile.dev/docs/tenant-virtualization/tenant-management
## **What is a tenant?**
A tenant is primarily a company, an organization, or a workspace in your product that contains a group of users. A SaaS application provides services to multiple tenants. Tenant is the basic building block of all SaaS applications. Every SaaS feature and experience is built on top of tenants. Nile has built-in tenant virtualization, which makes it easy, reliable, and cost-effective to develop and support SaaS use cases across the globe.
## Creating a tenant
Tenant is the basic building block for SaaS and Nile. Nile’s Postgres comes with a built-in tenant table. This table has built-in columns to support common use cases out of the box but can also be extended to add more application-specific columns.
```sql theme={null}
select * from tenants;
```
id
name
created
updated
deleted
The tenants table has an id that uniquely identifies a tenant, a name of the tenant (customer or workspace etc) and a few other columns for bookkeeping.
Inserting a new customer or tenant could use SQL, one of our SDKs or REST APIs. For product led growth companies, your application would typically dynamically create the new customer when they sign up using our SDK or REST APIs. If you are a more sales led product, you can use Nile’s tenant dashboard to add one.
```sql theme={null}
-- create a record for the first customer
insert into tenants (name) VALUES ('customer1');
```
You can see how a uuid is automatically inserted into the id column for the customer. This id can be used for all future reference to ‘customer1’.
```sql theme={null}
select * from tenants;
```
id
name
created
updated
deleted
018ac98e-b37a-731b-b03a-6617e8fd5266
customer1
2023-09-24 23:38:07.097633
2023-09-24 23:38:07.097633
The insert version shown here is the most simple option. You would be able to specify the location (any supported region in the world) or the deployment mode (place tenants on a dedicated instance) of a tenant for more fine grained control. You would also be able to extend the tenants table to add more metadata (covered in later sections).
## Adding data for each tenant
Once you have a tenant created, you can insert rows into different tables for a specific tenant. **You can create a tenant aware table in Nile by creating a table with a ‘tenant\_id’ column of type uuid.** tenant\_id is a reserved keyword. This is all it takes to ensure you get all the benefits of tenant virtualization in Nile’s Postgres.
For example, if you were building an employee management software that helps with managing employee information, benefits and payroll, it would have an ‘employees’ table to track employees data.
```sql theme={null}
-- creating an employee table that is tenant aware
create table employees (
tenant_id uuid,
id integer,
name text,
age integer,
address text,
start_date timestamp,
title text,
CONSTRAINT FK_tenants FOREIGN KEY(tenant_id) REFERENCES tenants(id),
CONSTRAINT PK_employee PRIMARY KEY(tenant_id,id));
```
```sql theme={null}
-- adding employees for customer 1
insert into employees (tenant_id, id, name, age, address, start_date, title)
values
('018ac98e-b37a-731b-b03a-6617e8fd5266',1345,'Jason',30,'Sunnyvale,California','2016-12-22 19:10:25-07','software engineer'),
('018ac98e-b37a-731b-b03a-6617e8fd5266',2423,'Minnie',24,'Seattle,Washingtom','2018-11-11 12:09:22-06','sales engineer'),
('018ac98e-b37a-731b-b03a-6617e8fd5266',4532,'Shiva',32,'Fremont, California','2019-09-05 04:03:12-05','product manager');
```
You can now repeat the same thing for another customer.
```sql theme={null}
-- create the second customer
insert into tenants (name) VALUES ('customer2');
select * from tenants;
```
id
name
created
updated
deleted
018ac98e-b37a-731b-b03a-6617e8fd5266
customer1
2023-09-24 23:38:07.097633
2023-09-24 23:38:07.097633
018aca35-b8c4-7674-882c-30ae56d7b479
customer2
2023-09-25 02:40:32.964067
2023-09-25 02:40:32.964067
```sql theme={null}
-- insert employees for the second customer
insert into employees (tenant_id, id, name, age, address, start_date, title)
values
('018aca35-b8c4-7674-882c-30ae56d7b479',5643,'John',36,'London,UK','2017-12-12 19:10:25-07','senior software engineer'),
('018aca35-b8c4-7674-882c-30ae56d7b479',1532,'Mark',27,'Manchester,UK','2022-10-10 12:09:22-06','support engineer'),
('018aca35-b8c4-7674-882c-30ae56d7b479',8645,'Sam',42,'Liverpool,UK','2015-08-04 04:03:12-05','product manager');
select * from employees;
```
tenant\_id
id
name
age
address
start\_date
title
018ac98e-b37a-731b-b03a-6617e8fd5266
1345
Jason
30
Sunnyvale,California
2016-12-22 19:10:25
software engineer
018ac98e-b37a-731b-b03a-6617e8fd5266
2423
Minnie
24
Seattle,Washington
2018-11-11 12:09:22
sales engineer
018ac98e-b37a-731b-b03a-6617e8fd5266
4532
Shiva
32
Fremont, California
2019-09-05 04:03:12
product manager
018aca35-b8c4-7674-882c-30ae56d7b479
5643
John
36
London,UK
2017-12-12 19:10:25
senior software engineer
018aca35-b8c4-7674-882c-30ae56d7b479
1532
Mark
27
Manchester,UK
2022-10-10 12:09:22
support engineer
018aca35-b8c4-7674-882c-30ae56d7b479
8645
Sam
42
Liverpool,UK
2015-08-04 04:03:12
product manager
## Querying tenant data
SaaS applications only query a specific tenant data at any given of time. This requires isolating data to ensure user of one tenant does not read the data of another tenant. You can do this in Nile’s Postgres by simply using the nile.tenant\_id session parameter in SQL or directly using our SDK. A good way to think about this is that setting the session context to a tenant is equivalent to pointing to a specific tenant DB. This ensures that all queries are directed to that specific tenant DB. At the same time, you still have access to shared tables (see shared table section).
```sql theme={null}
-- set the session context to a specific tenant
-- who needs to be isolated.
set nile.tenant_id = '018ac98e-b37a-731b-b03a-6617e8fd5266';
select * from employees
```
tenant\_id
id
name
age
address
start\_date
title
018ac98e-b37a-731b-b03a-6617e8fd5266
1345
Jason
30
Sunnyvale,California
2016-12-22 19:10:25
software engineer
018ac98e-b37a-731b-b03a-6617e8fd5266
2423
Minnie
24
Seattle,Washington
2018-11-11 12:09:22
sales engineer
018ac98e-b37a-731b-b03a-6617e8fd5266
4532
Shiva
32
Fremont, California
2019-09-05 04:03:12
product manager
## Extending tenant information
While Nile’s Postgres has a built in table to store tenants information, it is not strongly opinionated and lets you extend the table to add more columns (read about built in tables). This lets you to store additional metadata for a tenant. You can start storing the address and phone number of each tenant by adding those columns to the tenants table.
```sql theme={null}
ALTER TABLE tenants
ADD phone_number text,
ADD address text;
```
## Foreign keys across tenant tables
Extending the employee management SaaS example, we can add information to track the benefits plan that each employee has opted. The ‘benefits’ table has the plan that each employee has opted for and the cost of the plan per year. The employee id is a foreign key reference to the id in the employees table. All the standard behavior of foreign keys are applicable. You can read more about Foreign keys in the official [Postgres docs](https://www.postgresql.org/docs/current/tutorial-fk.html).
```sql theme={null}
create table benefits (
tenant_id uuid,
id integer,
employee_id integer,
plan_name text,
expiry_date timestamp,
cost integer,
CONSTRAINT FK_tenants FOREIGN KEY(tenant_id) REFERENCES tenants(id),
CONSTRAINT FK_employees FOREIGN KEY(tenant_id,employee_id) REFERENCES employees(tenant_id,id),
CONSTRAINT PK_benefits PRIMARY KEY(tenant_id,id));
```
Note in this example that the foreign key is for the composite tenant\_id,employee\_id pair. This is because you can have the same employee id across two different tenants or customers.
## Transactions for tenants
A common use case would be for SaaS applications to update multiple tables of the same tenant transactionally. For example, you may want to add a new employee record to the ‘employees’ table and their benefits to the ‘benefits’ table transactionally for a specific customer.
```sql theme={null}
BEGIN;
insert into employees (tenant_id, id, name, age, address, start_date, title)
values
('018aca35-b8c4-7674-882c-30ae56d7b479',2354,'Adam',22,'Manchester,UK','2016-11-11 12:10:15-08','Designer');
insert into benefits (tenant_id, id, employee_id, plan_name, expiry_date, cost)
values
('018aca35-b8c4-7674-882c-30ae56d7b479',1,2354,'Plan A', '2024-11-11 12:10:15-08',1000);
COMMIT;
```
## Deleting tenants
Deleting tenants would typically be used when a customer/tenant deletes their account. You may also want to delete a tenant to save cost when a customer churns due to no activity in the application. The behavior of deleting a tenant would be schema dependent. If the schema has cascading set, all the tenants data will be deleted when a tenant is removed from the tenants table.
```sql theme={null}
delete from tenants where id='018aca35-b8c4-7674-882c-30ae56d7b479';
```
# Tenant Placement
Source: https://thenile.dev/docs/tenant-virtualization/tenant-placement
One of the key benefits of tenant virtualization in Nile’s Postgres is that you can decide on placement strategies for individual tenants instead of an entire database. While tenants are placed differently, you would want the clients to work seamlessly and route correctly. Placements are of two types
1. Regional placement. You may want to place individual tenants (customers) in different regions in the world for compliance or latency reasons.
2. Infrastructure placement. You may want to place tenants in a multitenant or dedicated infrastructure. The decision for this will depend on the customer needs, cost and level of isolation needed.
Nile plans to support both types of placements. This feature is in progress and we would love to hear from you on our [Github discussion forum](https://github.com/orgs/niledatabase/discussions) or [Discord community](https://discord.gg/8UuBB84tTy).
The documentation here is subjected to change based on user feedback but the capability to have fine grained control over tenants will be supported in Nile’s Postgres.
## Regional placement
We wanted to make global placements really easy while abstracting all the hard parts to manage them. We would support the ability to locate a tenant in any location globally while creating the tenant. We hope to also support the ability to move tenants from one location to another with a simple update statement to the region of the tenant. This will probably come after we support placement during creation. The Nile SDK will seamlessly route to where the tenant is located. The shared tables (check 'Sharing Tenants' section) will be accessible across all the tenants irrespective of where the tenant is located.
```sql theme={null}
insert into tenants (name, region)
values ('customer 1', 'aws-us-east1');
insert into tenants (name, region)
values ('customer 2', 'aws-eu-west1');
```
## Infrastructure placement
A very common pattern with SaaS is to control what infrastructure are specific customers placed. There are many reasons to do this. A customer might explicitly want their data to be placed in an isolated database for security reasons, the tenant could be a critical customer for the business or your isolation strategy could be to simple place every tenant in a dedicated database.
Placing a tenant on a dedicated mode. The default instance would be used in this case.
```sql theme={null}
insert into tenants (name, region, deployment_mode)
values ('customer 2', 'aws-us-east1', 'dedicated');
```
Given dedicated may not be instantly provisioned in all cases, the status field would indicate if the tenant is ready to be used.
```sql theme={null}
select * from tenants;
```
id
name
region
deployment\_mode
status
created
updated
deleted
018ac98e-b37a-731b-b03a-6617e8fd5266
customer1
aws-us-east1
dedicated
provisioning
2023-09-24 23:38:07.097633
2023-09-24 23:38:07.097633
The tenant would be ready after a few seconds or minutes.
```sql theme={null}
select * from tenants;
```
id
name
region
deployment\_mode
status
created
updated
deleted
018ac98e-b37a-731b-b03a-6617e8fd5266
customer1
aws-us-east1
dedicated
ready
2023-09-24 23:38:07.097633
2023-09-24 23:38:07.097633
There would also be a way to specify specific instance type for a tenant in dedicated mode. This is still very early in development and we would love your thoughts or your use case on our [Github discussion forum](https://github.com/orgs/niledatabase/discussions) or our [Discord developer community](https://discord.com/invite/8UuBB84tTy).
# Tenant Sharing
Source: https://thenile.dev/docs/tenant-virtualization/tenant-sharing
# How do you share data between tenants?
While Nile provides a native approach to isolating data between tenants, it also supports sharing data across tenants using shared tables.
For example
1. A corporate travel booking site like TripActions ([https://www.navan.com/](https://www.navan.com/)) or Concur helps company employees book flights and manage their trips. Every company that uses the product must see a list of available flights. The list of flights can be stored in a shared table accessible to all tenants in any location.
2. A CRM product like Salesforce or Hubspot lists companies by categories that Sales reps can choose from for outreach. The list of companies and their categories is a global list that can be made available to all customers using shared tables.
3. Your application is an Infrastructure SaaS product (e.g., Snowflake or Databricks) and wants to provide a list of cloud providers and regions for users to select where they want to deploy their infrastructure. The data about all the cloud providers and the available regions can be in a shared table, and all the tenants can access it.
## Creating and inserting data into shared tables
To create a shared table, you must create a standard table without a tenant\_id column. Nile ensures that the data in the shared table is available across all tenants.
```sql theme={null}
create table flights (
id integer PRIMARY KEY,
name text,
from_location text,
to_location text,
departure_time TIMESTAMP,
arrival_time TIMESTAMP);
```
A table with a list of flights for a corporate travel booking site. The list of flights can be shared across tenants.
Inserting into shared table works as normal with SQL (TBD JS)
```sql theme={null}
insert into flights
(id, name, from_location, to_location, departure_time, arrival_time)
values
(643, 'United','SFO','NY','2023-12-22 19:10:25-07','2016-12-23 05:10:25-07'),
(123, 'Indian Airways','Mumbai','Delhi','2024-11-23 19:10:25-07','2024-11-23 01:10:25-07'),
(384, 'Singapore Airlines','SGR','LDN','2024-10-23 04:10:25-07','2024-11-23 10:10:25-07'),
(724, 'British Airways','LDN','Paris','2024-07-04 11:10:25-07','2024-07-04 17:10:25-07');
```
Most use cases for shared tables will involve loading data from some external source instead of manual inserts into the table. For example, to populate the flights table, you will typically load the data from an airline data company (eg. [Sabre](https://www.sabre.com/products/suites/developer-experience/sabre-apis/)) and also keep it upto date by subscribing to real time changes.
## Querying data from shared tables
Since shared tables are just standard Postgres tables, all types of queries work on Shared tables.
Selecting the flights that start from SFO
```sql theme={null}
select * from flights where from_location='SFO';
```
id
name
from\_location
to\_location
departure\_time
arrival\_time
643
United
SFO
NY
2023-12-22 19:10:25
2016-12-23 05:10:25
You can also query by joining multiple shared tables. We can join the “flights” table with “airports” table to get more detailed information about the destination from where the flights are leaving.
```sql theme={null}
-- the airports table where each entry represent a unique airport in the world
create table airports (
id integer PRIMARY KEY,
name text,
code text,
address text,
number_of_terminals integer);
-- entry of a few airports in the world with detailed information for each
insert into airports (id, name, code, address, number_of_terminals)
values
(3459,'San Francisco','SFO','San Francisco, CA 94128','10'),
(2933,'Singapore','SGR','PO Box 168 Changi Airport Singapore 918146','15'),
(2234,'London','LDN','Hounslow TW6 1QG, UK','12');
-- displaying all the flight information along with the address of the airport
-- from which the flight is starting
select flights.id, flights.name, flights.from_location, airports.address
from flights, airports
where flights.from_location=airports.code;
```
id
name
from\_location
address
643
United
SFO
San Francisco, CA 94128
## Joining data between tenant aware and shared tables
Tenant aware tables can be joined with shared tables to provide relevant information for a specific tenant. For example, let us consider a bookings table that has all the flights booked by employees of a specific company(or tenant) with id **7543a610-f068-45fa-8ba0-7bc884bd29c2**. Each booking will refer to the flights id in the ‘flights’ table. It would be trivial to show detailed flight information along with every booking by joining the two tables.
```sql theme={null}
-- the bookings table where each row represents a single booking
-- for a specific employee within a customer/tenant
create table bookings (
tenant_id uuid,
booking_id integer,
employee_id text,
flight_id integer,
total_price float,
PRIMARY KEY(tenant_id,booking_id));
-- inserting rows for all the bookings done by employees in a
-- specific tenant
insert into bookings (tenant_id, booking_id, employee_id, flight_id, total_price)
values
('7543a610-f068-45fa-8ba0-7bc884bd29c2',1345,2,643, 200.00),
('7543a610-f068-45fa-8ba0-7bc884bd29c2',2456,3,123, 150.00),
('7543a610-f068-45fa-8ba0-7bc884bd29c2',3240,4,123, 250.00);
-- set tenant context to ensure selects are isolated only to this tenant
set nile.tenant_id = '7543a610-f068-45fa-8ba0-7bc884bd29c2';
-- join query between bookings and flights to display each booking info
-- in the application with detailed information about the flight
select bookings.booking_id, bookings.employee_id, flights.name,
flights.from_location, flights.to_location, flights.departure_time,
flights.arrival_time, bookings.total_price
from bookings,flights
where bookings.flight_id=flights.id;
```
booking\_id
employee\_id
name
from\_destination
to\_destination
departure\_time
arrival\_time
total\_price
1345
2
United
SFO
NY
2023-12-22 19:10:25
2016-12-23 05:10:25
200.00
2456
3
Indian Airways
Mumbai
Delhi
2024-11-23 19:10:25
2024-11-23 01:10:25
150.00
3240
4
Indian Airways
Mumbai
Delhi
2024-11-23 19:10:25
2024-11-23 01:10:25
250.00
## Foreign keys for shared tables
A shared table can have a foreign key reference to another shared table. In the travel example, the flights table can have a reference to the airport id. So, the ‘flights’ table can be modified to have an airport id column that is a foreign reference to the id column in the ‘airports’ table
```sql theme={null}
-- flights table's airport_id is a reference to the id column
-- in table airports
create table flights (
id integer PRIMARY KEY,
airport_id integer,
name text,
from_location text,
to_location text,
departure_time TIMESTAMP,
arrival_time TIMESTAMP,
**CONSTRAINT FK_airport_id FOREIGN KEY(airport_id) REFERENCES airports(id)**
);
```
## Transactions with shared tables
Finally, it is possible to do writes to multiple shared tables in the same transaction. A good use case for this is to ensure multiple shared tables are in a consistent state when they are loaded with data. While the example above showed that the flights and airport tables were populated separately, another alternative is to ensure that for each airport that is inserted in the ‘airports’ table, all the flights are updated in the ‘flights’ table. This is specifically useful when data is bootstrapped from some external source.
```sql theme={null}
BEGIN;
insert into airports (id, name, code, address, number_of_terminals)
values
(4956,'Seattle','SEA','17801 International Blvd','15');
insert into flights
(id, name, from_location, to_location, departure_time, arrival_time)
values
(942, 'Alaskan','SEA','SFO','2024-12-22 11:10:25-07','2024-12-22 04:10:25-07'),
(342, 'United','SEA','PHX','2024-10-22 16:10:25-07','2024-10-22 18:10:25-07');
COMMIT;
```