deno_kv_fs
[!IMPORTANT]
PLEASE GIVE A STAR ⭐
Deno KV file system, compatible with Deno deploy. Saves files in 64kb chunks. You can organize files into directories. You can control the KB/s rate for saving and reading files, rate limit, user space limit and limit concurrent operations, useful for controlling uploads/downloads. Makes use of Web Streams API.
Contents
How to use
Instantiating the class:
const kvFs = new DenoKvFs();
//If you want to use your existing instance of Deno.Kv
const myDenoKV = await Deno.openKv(/* your parameters */);
const kvFs = new DenoKvFs(myDenoKV);
Accepts any stream as input, and returns a generic stream of type: "bytes"
.
Files that are saved incompletely are automatically deleted. Read methods return the processing status of a file (if it is currently being processed). This is useful for knowing the progress status of a save/update/delete. If a file does not exist, null is returned.
The method save
is used to save files and has the following interface as input
parameter:
interface SaveOptions {
path: string[]; //Mandatory. The root directory is []
content: ReadableStream | Uint8Array | string; //Mandatory
chunksPerSecond?: number;
clientId?: string | number;
validateAccess?: (path: string[]) => Promise<boolean> | boolean;
maxClientIdConcurrentReqs?: number;
maxFileSizeBytes?: number;
allowedExtensions?: string[];
}
The read
, readDir
, delete
and deleteDir
methods are intended to read and
delete files, and have the following interface as input parameters:
interface ReadOptions {
path: string[]; //Mandatory
chunksPerSecond?: number;
maxDirEntriesPerSecond?: number;
clientId?: string | number;
validateAccess?: (path: string[]) => Promise<boolean> | boolean;
maxClientIdConcurrentReqs?: number;
pagination?: boolean; //If pagination is true, will return the cursor to the next page (if exists).
cursor?: string; //for readDir, If there is a next page.
}
Examples
Saving data
import { toReadableStream } from "jsr:@std/io";
const fileName = "myFile.txt";
let resData = await kvFs.save({
path: ["my_dir", fileName],
content: toReadableStream(await Deno.open(fileName)),
});
Saving data directly
Isso uses a Uint8Array
or string
as file content. This is not recommended,
It can fill up your RAM memory, use only for internal resources of your
application. For optimized use, use an instance of ReadableStream
.
const fileName = "myFile.txt";
let resData = await kvFs.save({
path: ["my_dir", fileName],
content: await Deno.readFile(fileName), //Or content: "myStringData"
});
Saving data from a submitted form
const reqBody = await request.formData();
const existingFileNamesInTheUpload: { [key: string]: number } = {};
const res: any = {};
for (const item of reqBody.entries()) {
if (item[1] instanceof File) {
const formField: any = item[0];
const fileData: any = item[1];
if (!existingFileNamesInTheUpload[fileData.name]) {
existingFileNamesInTheUpload[fileData.name] = 1;
} else {
existingFileNamesInTheUpload[fileData.name]++;
}
let prepend = "";
if (existingFileNamesInTheUpload[fileData.name] > 1) {
prepend += existingFileNamesInTheUpload[fileData.name].toString();
}
const fileName = prepend + fileData.name;
let resData = await kvFs.save({
path: ["my_dir", fileName],
content: fileData.stream(),
});
if (res[formField] !== undefined) {
if (Array.isArray(res[formField])) {
res[formField].push(resData);
} else {
res[formField] = [res[formField], resData];
}
} else {
res[formField] = resData;
}
}
}
console.log(res);
In frontend
<form
id="yourFormId"
enctype="multipart/form-data"
action="/upload"
method="post"
>
<input type="file" name="file1" multiple />
<br />
<input type="submit" value="Submit" />
</form>
<script>
var files = document.querySelector("#yourFormId input[type=file]").files;
var name = document.querySelector("#yourFormId input[type=file]")
.getAttribute(
"name",
);
var form = new FormData();
for (var i = 0; i < files.length; i++) {
form.append(`${name}_${i}`, files[i]);
}
var res = await fetch(`/your_POST_URL`, { //Fetch API automatically puts the form in the format "multipart/form-data".
method: "POST",
body: form,
}).then((response) => response.json());
console.log(res);
</script>
Returning data
const fileName = "myFile.txt";
let resData = await kvFs.read({
path: ["my_dir", fileName],
});
response.body = resData.content; //resData.content is an instance of ReadableStream
Returning data directly
This returns the file content as a Uint8Array
or string
. This is not
recommended, it can fill up your RAM memory, use only for internal resources
of your application. For optimized use, use the ReadableStream
=>
type: "bytes"
that comes by default in file.content
.
const fileName = "myFile.txt";
let resData = await kvFs.read({
path: ["my_dir", fileName],
});
response.body = await DenoKvFs.readStream(resData.content); //Or await DenoKvFs.readStreamAsString(resData.content)
Example of a function to control data traffic
const gigabyte = 1024 * 1024 * 1024;
const existingRequests = kvFs.getClientReqs(user.id); //The input parameter is the same as clientId
const chunksPerSecond = (user.isPremium() ? 20 : 1) / existingRequests;
const maxClientIdConcurrentReqs = user.isPremium() ? 5 : 1;
const maxFileSizeBytes = (user.isPremium() ? 1 : 0.1) * gigabyte;
//To read
let resData = await kvFs.read({
path: ["my_dir", fileName],
chunksPerSecond: chunksPerSecond,
maxClientIdConcurrentReqs: maxClientIdConcurrentReqs,
maxFileSizeBytes: maxFileSizeBytes,
clientId: user.id, //The clientId can also be the remote address of a request, for example.
});
response.body = resData.content;
//Delete
let resData = await kvFs.delete({
path: ["my_dir_2", fileName],
chunksPerSecond: chunksPerSecond,
maxClientIdConcurrentReqs: maxClientIdConcurrentReqs,
maxFileSizeBytes: maxFileSizeBytes,
clientId: user.id,
});
//Read dir
const maxDirEntriesPerSecond = user.isPremium() ? 1000 : 100;
let resData = await kvFs.readDir({
path: ["my_dir"],
chunksPerSecond: chunksPerSecond,
maxClientIdConcurrentReqs: maxClientIdConcurrentReqs,
maxFileSizeBytes: maxFileSizeBytes,
clientId: user.id, //The clientId can also be the remote address of a request, for example.
maxDirEntriesPerSecond: maxDirEntriesPerSecond,
pagination: true, //each page has 1000 entries
cursor: "JDhiasgPh", //If exists
});
//Delete dir
let resData = await kvFs.deleteDir({
path: ["my_dir"],
chunksPerSecond: chunksPerSecond,
maxClientIdConcurrentReqs: maxClientIdConcurrentReqs,
maxFileSizeBytes: maxFileSizeBytes,
clientId: user.id, //The clientId can also be the remote address of a request, for example.
maxDirEntriesPerSecond: maxDirEntriesPerSecond,
});
//Controlling maximum user space
const maxAvailableSpace = (user.isPremium() ? 1 : 0.1) * gigabyte;
let dirList = await kvFs.readDir({ //You can also set limits here
path: [user.id, "files"], //example
});
if (dirList.size > maxAvailableSpace) {
throw new Error(
`You have exceeded the ${maxAvailableSpace} GB limit of available space.`,
);
}
//validate access
let resData = await kvFs.readDir({
path: ["my_dir"],
validateAccess: async (path: string[]) =>
user.hasDirAccess(path) ? true : false,
});
Sending file progress in real time
kvFs.onFileProgress(status:FileStatus) => webSocket.send(JSON.stringify(status))
Useful procedures included
static async readStream(stream: ReadableStream): Promise<Uint8Array>
static async readStreamAsString(stream: ReadableStream): Promise<string>
getClientReqs(clientId: string | number): number
getAllFilesStatuses(): FileStatus[]
pathToURIComponent(path: string[]): string
URIComponentToPath(path: string): string[]
async save(options: SaveOptions): Promise<FileStatus | File>
async read(options: ReadOptions): Promise<File | FileStatus | null>
async readDir(options: ReadOptions): Promise<DirList>
async delete(options: ReadOptions): Promise<void | FileStatus>
async deleteDir(options: ReadOptions): Promise<FileStatus[]>
All imports
import {
DenoKvFs,
DirList,
File,
FileStatus,
ReadOptions,
SaveOptions,
} from "https://deno.land/x/deno_kv_fs/mod.ts";
About
Author: Henrique Emanoel Viana, a Brazilian computer scientist, enthusiast of web technologies, cel: +55 (41) 99999-4664. URL: https://sites.google.com/view/henriqueviana
Improvements and suggestions are welcome!