deno_kv_fs

[!IMPORTANT]
PLEASE GIVE A STAR ⭐

Deno KV file system, compatible with Deno deploy. Saves files in 64kb chunks. You can organize files into directories. You can control the KB/s rate for saving and reading files, rate limit, user space limit and limit concurrent operations, useful for controlling uploads/downloads. Makes use of Web Streams API.

Contents

How to use

Instantiating the class:

const kvFs = new DenoKvFs();
//If you want to use your existing instance of Deno.Kv
const myDenoKV = await Deno.openKv(/* your parameters */);
const kvFs = new DenoKvFs(myDenoKV);

Accepts any stream as input, and returns a generic stream of type: "bytes".

Files that are saved incompletely are automatically deleted. Read methods return the processing status of a file (if it is currently being processed). This is useful for knowing the progress status of a save/update/delete. If a file does not exist, null is returned.

The method save is used to save files and has the following interface as input parameter:

interface SaveOptions {
  path: string[]; //Mandatory. The root directory is []
  content: ReadableStream | Uint8Array | string; //Mandatory
  chunksPerSecond?: number;
  clientId?: string | number;
  validateAccess?: (path: string[]) => Promise<boolean> | boolean;
  maxClientIdConcurrentReqs?: number;
  maxFileSizeBytes?: number;
  allowedExtensions?: string[];
}

The read, readDir, delete and deleteDir methods are intended to read and delete files, and have the following interface as input parameters:

interface ReadOptions {
  path: string[]; //Mandatory
  chunksPerSecond?: number;
  maxDirEntriesPerSecond?: number;
  clientId?: string | number;
  validateAccess?: (path: string[]) => Promise<boolean> | boolean;
  maxClientIdConcurrentReqs?: number;
  pagination?: boolean; //If pagination is true, will return the cursor to the next page (if exists).
  cursor?: string; //for readDir, If there is a next page.
}

Examples

Saving data

import { toReadableStream } from "jsr:@std/io";
const fileName = "myFile.txt";
let resData = await kvFs.save({
  path: ["my_dir", fileName],
  content: toReadableStream(await Deno.open(fileName)),
});

Saving data directly

Isso uses a Uint8Array or string as file content. This is not recommended, It can fill up your RAM memory, use only for internal resources of your application. For optimized use, use an instance of ReadableStream.

const fileName = "myFile.txt";
let resData = await kvFs.save({
  path: ["my_dir", fileName],
  content: await Deno.readFile(fileName), //Or content: "myStringData"
});

Saving data from a submitted form

const reqBody = await request.formData();
const existingFileNamesInTheUpload: { [key: string]: number } = {};
const res: any = {};
for (const item of reqBody.entries()) {
  if (item[1] instanceof File) {
    const formField: any = item[0];
    const fileData: any = item[1];
    if (!existingFileNamesInTheUpload[fileData.name]) {
      existingFileNamesInTheUpload[fileData.name] = 1;
    } else {
      existingFileNamesInTheUpload[fileData.name]++;
    }
    let prepend = "";
    if (existingFileNamesInTheUpload[fileData.name] > 1) {
      prepend += existingFileNamesInTheUpload[fileData.name].toString();
    }
    const fileName = prepend + fileData.name;
    let resData = await kvFs.save({
      path: ["my_dir", fileName],
      content: fileData.stream(),
    });
    if (res[formField] !== undefined) {
      if (Array.isArray(res[formField])) {
        res[formField].push(resData);
      } else {
        res[formField] = [res[formField], resData];
      }
    } else {
      res[formField] = resData;
    }
  }
}
console.log(res);

In frontend

<form
  id="yourFormId"
  enctype="multipart/form-data"
  action="/upload"
  method="post"
>
  <input type="file" name="file1" multiple />
  <br />
  <input type="submit" value="Submit" />
</form>
<script>
  var files = document.querySelector("#yourFormId input[type=file]").files;
  var name = document.querySelector("#yourFormId input[type=file]")
    .getAttribute(
      "name",
    );
  var form = new FormData();
  for (var i = 0; i < files.length; i++) {
    form.append(`${name}_${i}`, files[i]);
  }
  var res = await fetch(`/your_POST_URL`, { //Fetch API automatically puts the form in the format "multipart/form-data".
    method: "POST",
    body: form,
  }).then((response) => response.json());
  console.log(res);
</script>

Returning data

const fileName = "myFile.txt";
let resData = await kvFs.read({
  path: ["my_dir", fileName],
});
response.body = resData.content; //resData.content is an instance of ReadableStream

Returning data directly

This returns the file content as a Uint8Array or string. This is not recommended, it can fill up your RAM memory, use only for internal resources of your application. For optimized use, use the ReadableStream => type: "bytes" that comes by default in file.content.

const fileName = "myFile.txt";
let resData = await kvFs.read({
  path: ["my_dir", fileName],
});
response.body = await DenoKvFs.readStream(resData.content); //Or await DenoKvFs.readStreamAsString(resData.content)

Example of a function to control data traffic

const gigabyte = 1024 * 1024 * 1024;
const existingRequests = kvFs.getClientReqs(user.id); //The input parameter is the same as clientId
const chunksPerSecond = (user.isPremium() ? 20 : 1) / existingRequests;
const maxClientIdConcurrentReqs = user.isPremium() ? 5 : 1;
const maxFileSizeBytes = (user.isPremium() ? 1 : 0.1) * gigabyte;

//To read
let resData = await kvFs.read({
  path: ["my_dir", fileName],
  chunksPerSecond: chunksPerSecond,
  maxClientIdConcurrentReqs: maxClientIdConcurrentReqs,
  maxFileSizeBytes: maxFileSizeBytes,
  clientId: user.id, //The clientId can also be the remote address of a request, for example.
});
response.body = resData.content;
//Delete
let resData = await kvFs.delete({
  path: ["my_dir_2", fileName],
  chunksPerSecond: chunksPerSecond,
  maxClientIdConcurrentReqs: maxClientIdConcurrentReqs,
  maxFileSizeBytes: maxFileSizeBytes,
  clientId: user.id,
});

//Read dir
const maxDirEntriesPerSecond = user.isPremium() ? 1000 : 100;
let resData = await kvFs.readDir({
  path: ["my_dir"],
  chunksPerSecond: chunksPerSecond,
  maxClientIdConcurrentReqs: maxClientIdConcurrentReqs,
  maxFileSizeBytes: maxFileSizeBytes,
  clientId: user.id, //The clientId can also be the remote address of a request, for example.
  maxDirEntriesPerSecond: maxDirEntriesPerSecond,
  pagination: true, //each page has 1000 entries
  cursor: "JDhiasgPh", //If exists
});

//Delete dir
let resData = await kvFs.deleteDir({
  path: ["my_dir"],
  chunksPerSecond: chunksPerSecond,
  maxClientIdConcurrentReqs: maxClientIdConcurrentReqs,
  maxFileSizeBytes: maxFileSizeBytes,
  clientId: user.id, //The clientId can also be the remote address of a request, for example.
  maxDirEntriesPerSecond: maxDirEntriesPerSecond,
});

//Controlling maximum user space
const maxAvailableSpace = (user.isPremium() ? 1 : 0.1) * gigabyte;
let dirList = await kvFs.readDir({ //You can also set limits here
  path: [user.id, "files"], //example
});
if (dirList.size > maxAvailableSpace) {
  throw new Error(
    `You have exceeded the ${maxAvailableSpace} GB limit of available space.`,
  );
}

//validate access
let resData = await kvFs.readDir({
  path: ["my_dir"],
  validateAccess: async (path: string[]) =>
    user.hasDirAccess(path) ? true : false,
});

Sending file progress in real time

kvFs.onFileProgress(status:FileStatus) => webSocket.send(JSON.stringify(status))

Useful procedures included

  • static async readStream(stream: ReadableStream): Promise<Uint8Array>
  • static async readStreamAsString(stream: ReadableStream): Promise<string>
  • getClientReqs(clientId: string | number): number
  • getAllFilesStatuses(): FileStatus[]
  • pathToURIComponent(path: string[]): string
  • URIComponentToPath(path: string): string[]
  • async save(options: SaveOptions): Promise<FileStatus | File>
  • async read(options: ReadOptions): Promise<File | FileStatus | null>
  • async readDir(options: ReadOptions): Promise<DirList>
  • async delete(options: ReadOptions): Promise<void | FileStatus>
  • async deleteDir(options: ReadOptions): Promise<FileStatus[]>

All imports

import {
  DenoKvFs,
  DirList,
  File,
  FileStatus,
  ReadOptions,
  SaveOptions,
} from "https://deno.land/x/deno_kv_fs/mod.ts";

About

Author: Henrique Emanoel Viana, a Brazilian computer scientist, enthusiast of web technologies, cel: +55 (41) 99999-4664. URL: https://sites.google.com/view/henriqueviana

Improvements and suggestions are welcome!