deno_kv_fs

Deno KV file system, compatible with Deno deploy. Saves files in 64kb chunks. You can organize files into directories. You can control the KB/s rate for saving and reading files, rate limit, user space limit and limit concurrent operations, useful for controlling uploads/downloads. Makes use of Web Streams API.

Contents

How to use

Instantiating the class:

const kvFs = new DenoKvFs();
//If you want to use your existing instance of Deno.Kv
const myDenoKV = await Deno.openKv(/* your parameters */);
const kvFs = new DenoKvFs(myDenoKV);

All streams used by the library are type: "bytes".

Files that are saved incompletely are automatically deleted. Read methods return the processing status of a file (if it is currently being processed). This is useful for knowing the progress status of a save/update/delete. If a file does not exist, null is returned.

The method save is used to save files and has the following interface as input parameter:

interface SaveOptions {
  path: string[]; //Mandatory
  content: ReadableStream<Uint8Array> | Uint8Array | string; //Mandatory
  chunksPerSecond?: number;
  clientId?: string | number;
  maxClientIdConcurrentReqs?: number;
  maxFileSizeBytes?: number;
  allowedExtensions?: string[];
}

The read, readDir, delete and deleteDir methods are intended to read and delete files, and have the following interface as input parameters:

interface ReadOptions {
  path: string[]; //Mandatory
  chunksPerSecond?: number;
  maxDirEntriesPerSecond?: number;
  clientId?: string | number;
  maxClientIdConcurrentReqs?: number;
}

Examples

Saving data from a file

import { toReadableStream } from "jsr:@std/io";
const fileName = "myFile.txt";
let resData = await kvFs.save({
  path: ["my_dir", fileName],
  content: toReadableStream(await Deno.open(fileName)),
});

Saving data directly

Isso uses a Uint8Array or string as file content. This is not recommended, use only for internal resources of your application. For optimized use, use an instance of ReadableStream => type: "bytes".

import { toReadableStream } from "jsr:@std/io";
const fileName = "myFile.txt";
let resData = await kvFs.save({
  path: ["my_dir", fileName],
  content: await Deno.readFile(fileName), //Or content: "myStringData"
});

Saving data from a submitted form

const reqBody = await request.formData();
const existingFileNamesInTheUpload: { [key: string]: number } = {};
const res: any = {};
for (const item of reqBody.entries()) {
  if (item[1] instanceof File) {
    const formField: any = item[0];
    const fileData: any = item[1];
    if (!existingFileNamesInTheUpload[fileData.name]) {
      existingFileNamesInTheUpload[fileData.name] = 1;
    } else {
      existingFileNamesInTheUpload[fileData.name]++;
    }
    let prepend = "";
    if (existingFileNamesInTheUpload[fileData.name] > 1) {
      prepend += existingFileNamesInTheUpload[fileData.name].toString();
    }
    const fileName = prepend + fileData.name;
    let resData = await kvFs.save({
      path: ["my_dir", fileName],
      content: fileData.stream(),
    });
    if (res[formField] !== undefined) {
      if (Array.isArray(res[formField])) {
        res[formField].push(resData);
      } else {
        res[formField] = [res[formField], resData];
      }
    } else {
      res[formField] = resData;
    }
  }
}
console.log(res);

In frontend

<form id="yourFormId" enctype="multipart/form-data" action="/upload" method="post">
  <input type="file" name="file1" multiple><br>
  <input type="submit" value="Submit">
</form>
<script>
  var files = document.querySelector("#yourFormId input[type=file]").files;
  var name = document.querySelector("#yourFormId input[type=file]").getAttribute(
    "name",
  );
  var form = new FormData();
  for (var i = 0; i < files.length; i++) {
    form.append(`${name}_${i}`, files[i]);
  }
  var res = await fetch(`/your_POST_URL`, { //Fetch API automatically puts the form in the format "multipart/form-data".
    method: "POST",
    body: form,
  }).then((response) => response.json());
  console.log(res);
</script>

Returning data

const fileName = "myFile.txt";
let resData = await kvFs.read({
  path: ["my_dir", fileName],
});
response.body = resData.content; //resData.content is an instance of ReadableStream

Returning data directly

This returns the file content as a Uint8Array or string. This is not recommended, use only for internal resources of your application. For optimized use, use the ReadableStream => type: "bytes" that comes by default.

const fileName = "myFile.txt";
let resData = await kvFs.read({
  path: ["my_dir", fileName],
});
response.body = await DenoKvkvFs.readStream(resData.content); //Or await DenoKvkvFs.readStreamAsString(resData.content)

Example of a function to control data traffic

const gigabyte = 1024 * 1024 * 1024;
const existingRequests = kvFs.getClientReqs(user.id); //The input parameter is the same as clientId
const chunksPerSecond = (user.isPremium() ? 20 : 1) / existingRequests;
const maxClientIdConcurrentReqs = user.isPremium() ? 5 : 1;
const maxFileSizeBytes = (user.isPremium() ? 1 : 0.1) * gigabyte;

let resData = await kvFs.read({
  path: ["my_dir", fileName],
  chunksPerSecond: chunksPerSecond,
  maxClientIdConcurrentReqs: maxClientIdConcurrentReqs,
  maxFileSizeBytes: maxFileSizeBytes,
  clientId: user.id, //The clientId can also be the remote address of a request, for example.
});
response.body = resData.content;

Limiting the space available to a user

const gigabyte = 1024 * 1024 * 1024;
const maxAvailableSpace = (user.isPremium() ? 1 : 0.1) * gigabyte;

const maxDirEntriesPerSecond = user.isPremium() ? 1000 : 100;

let dirList = await kvFs.readDir({
  maxDirEntriesPerSecond: maxDirEntriesPerSecond,
  path: [user.id, "files"], //example
});
if (dirList.size > maxAvailableSpace) {
  throw new Error(
    `You have exceeded the ${maxAvailableSpace} GB limit of available space.`,
  );
}

Sending file progress in real time

kvFs.onFileProgress(status:FileStatus) => webSocket.send(JSON.stringify(status))

Useful procedures included

  • static async readStream(stream: ReadableStream): Promise<Uint8Array>
  • static async readStreamAsString(stream: ReadableStream): Promise<string>
  • getClientReqs(clientId: string | number): number
  • getAllFilesStatuses(): FileStatus[]
  • pathToURIComponent(path: string[]): string
  • URIComponentToPath(path: string): string[]
  • async save(options: SaveOptions): Promise<FileStatus | File>
  • async read(options: ReadOptions): Promise<File | FileStatus | null>
  • async readDir(options: ReadOptions): Promise<DirList>
  • async delete(options: ReadOptions): Promise<void | FileStatus>
  • async deleteDir(options: ReadOptions): Promise<FileStatus[]>

All imports

import {
  DenoKvFs,
  DirList,
  File,
  FileStatus,
  ReadOptions,
  SaveOptions,
} from "https://deno.land/x/deno_kv_fs/mod.ts";

About

Author: Henrique Emanoel Viana, a Brazilian computer scientist, enthusiast of web technologies, cel: +55 (41) 99999-4664. URL: https://sites.google.com/view/henriqueviana

Improvements and suggestions are welcome!