README.md 30.2 KB
Newer Older
1
# JavaScript Database (JSDB)
Aral Balkan's avatar
Aral Balkan committed
2

3
__Work in progress:__ A transparent, in-memory, streaming write-on-update JavaScript database for Small Web applications that persists to a JavaScript transaction log.
Aral Balkan's avatar
Aral Balkan committed
4

5
6
__Needless to say, this is not ready for use yet. But feel free to take a look around.__

Aral Balkan's avatar
Aral Balkan committed
7
8
## Roadmap to version 1.0.0.

Aral Balkan's avatar
Aral Balkan committed
9
  - [x] __Implement persistence.__ (15 Sept)
Aral Balkan's avatar
Aral Balkan committed
10
11
12
  - [x]  ╰─ Add unit tests for persistence. (19 Sept)
  - [x]  ╰─ Document persistence. (19 Sept)
  - [x]  ╰─ Add persistence example. (19 Sept)
Aral Balkan's avatar
Aral Balkan committed
13
  - [x] __Implement queries.__ (22 Sept)
Aral Balkan's avatar
Aral Balkan committed
14
  - [x]  ╰─ Add queries example. (22 Sept)
Aral Balkan's avatar
Aral Balkan committed
15
16
17
18
  - [x] __Refactor to implement persistence as append-only JavaScript transaction log and use streaming writes.__ (29 Sept)
  - [x]  ╰─  Update documentation to reflect new persistence engine. (29 Sept)
  - [x]  ╰─  Update examples to work with new persistence engine. (30 Sept)
  - [x] __Continue working on queries.__ (1 Oct)
Aral Balkan's avatar
Aral Balkan committed
19
20
  - [x]  ╰─ Add unit tests for queries. (1 Oct)
  - [x]  ╰─ Document queries. (1 Oct)
Aral Balkan's avatar
Aral Balkan committed
21
  - [x] __Bring code coverage back up to 100%.__ (2 Oct)
Aral Balkan's avatar
Aral Balkan committed
22
  - [x] __Implement safety controls on instantiation and table replacement.__ (5 Oct)
23
  - [ ] __Implement JSDF serialiser__ (inc. support for custom objects, and Date, etc.) _(in progress)_
Aral Balkan's avatar
Aral Balkan committed
24
  - [ ] __Integrate into [Site.js](https://sitejs.org)__ _(in progress)_
Aral Balkan's avatar
Aral Balkan committed
25
  - [ ] __Use/test on upcoming small-web.org site__
26
27
  - [ ] __Release version 1.0.0__

28

29
30
## Ideas for post 1.0.0.

Aral Balkan's avatar
Aral Balkan committed
31
  - [ ] __Implement [transactions](https://github.com/small-tech/jsdb/issues/1).__
Aral Balkan's avatar
Aral Balkan committed
32
33
  - [ ]  ╰─ Ensure 100% code coverage for transactions.
  - [ ]  ╰─ Document transactions.
Aral Balkan's avatar
Aral Balkan committed
34
  - [ ]  ╰─ Add transaction example.
35
  - [ ] __Implement indices.__
Aral Balkan's avatar
Aral Balkan committed
36
37
38
  - [ ]  ╰─ Ensure 100% code coverage for indices.
  - [ ]  ╰─ Document indices.
  - [ ]  ╰─ Add indices example.
Aral Balkan's avatar
Aral Balkan committed
39

Aral Balkan's avatar
Aral Balkan committed
40

41
42
43
44
45
46
47
48
49
50
51
52
53
## Use case

A data layer for simple [Small Web](https://ar.al/2020/08/07/what-is-the-small-web/) sites for basic public (e.g., anonymous comments on articles) or configuration data. Built for use in [Site.js](https://sitejs.org).

__Not to farm people for their data.__ Surveillance capitalists can jog on now.


## Features

  - __Transparent:__ if you know how to work with arrays and objects and call methods in JavaScript, you already know how to use JSDB? It’s not called JavaScript Database for nothing.

  - __Automatic:__ it just works. No configuration.

Aral Balkan's avatar
Aral Balkan committed
54
55
  - __100% code coverage:__ meticulously tested. Note that this does not mean it is bug free ;)

56
57
58
59
60
61
62
63
64
65
66

## Limitations

  - __Small Data:__ this is for small data, not Big Data™.

  - __For Node.js:__ will not work in the browser. (Although data tables are plain JavaScript files and can be loaded in the browser.)

  - __Runs on untrusted nodes:__ this is for data kept on untrusted nodes (servers). Use it judiciously if you must for public data, configuration data, etc. If you want to store personal data or model human communication, consider end-to-end encrypted and peer-to-peer replicating data structures instead to protect privacy and freedom of speech. Keep an eye on the work taking place around the [Hypercore Protocol](https://hypercore-protocol.org/).

  - __In-memory:__ all data is kept in memory and, [without tweaks, cannot exceed 1.4GB in size](https://www.the-data-wrangler.com/nodejs-memory-limits/). While JSDB will work with large datasets, that’s not its primary purpose and it’s definitely not here to help you farm people for their data, so please don’t use it for that. (If that’s what you want, quite literally every other database out there is for your use case so please use one of those instead.)

Aral Balkan's avatar
Aral Balkan committed
67
  - __Streaming writes on update:__ writes are streamed to disk to an append-only transaction log as JavaScript statements and are both quick (in the single-digit miliseconds region on a development laptop with an SSD drive) and as safe as we can make them (synchronous as the kernel level).
68
69
70
71

  - __No schema, no migrations__: again, this is meant to be a very simple persistence, query, and observation layer for local server-side data. If you want schemas and migrations, take a look at nearly every other database out there.


72
73
## To install

Aral Balkan's avatar
Aral Balkan committed
74
75
76
77
78
Currently, you need to clone or install from this repo as this is a work-in-progress and no releases have been made yet to npm.

```
npm i github:small-tech/jsdb
```
79

80

81
82
83
84
85
## Usage

Here’s a quick example to whet your appetite:

```js
Aral Balkan's avatar
Aral Balkan committed
86
const JSDB = require('@small-tech/jsdb')
87
88
89

// Create your database in the test folder.
// (This is where your JSON files – “tables” – will be saved.)
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
//
const db = JSDB.open('db')

// Create db/people.js table with some initial data if it
// doesn’t already exist.
if (!db.people) {
  db.people = [
    {name: 'Aral', age: 43},
    {name: 'Laura', age: 34}
  ]

  // Correct Laura’s age. (This will automatically update db/people.js)
  db.people[1].age = 33

  // Add Oskar to the family. (This will automatically update db/people.js)
  db.people.push({name: 'Oskar', age: 8})

  // Update Oskar’s name to use his nickname. (This will automatically update db/people.js)
  db.people[2].name = 'Osky'
}
110
111
```

112
113
After running the above script, take a look at the resulting database table in the `./db/people.js` file.

114
115
## JavaScript Data Format (JSDF)

116
117
118
119
120
121
122
123
124
125
126
127
128
JSDB tables are written into JavaScript Data Format (JSDF) files. A JSDF file is a plain JavaScript file that comprises an append-only transaction log that creates the table in memory. For our example, it looks like this:

```js
globalThis._ = [];
(function () { if (typeof define === 'function' && define.amd) { define([], globalThis._); } else if (typeof module === 'object' && module.exports) { module.exports = globalThis._ } else { globalThis.people = globalThis._ } })();
_[0] = JSON.parse(`{"name":"Aral","age":43}`);
_[1] = JSON.parse(`{"name":"Laura","age":34}`);
_[1]['age'] = 33;
_[2] = JSON.parse(`{"name":"Oskar","age":8}`);
_['length'] = 3;
_[2]['name'] = `Osky`;
```

Aral Balkan's avatar
Aral Balkan committed
129
(Note: the format is a work-in-progress like the rest of the project at the moment. I’m considering cleaning up the superfluous length statements and weighing up the performance hit of maintaining state to enable that versus the potential use cases of a cleaner log – like history replay for example – and file size/initial load speed, which is really not too much of a concern given that they occur at server start for our use cases).
130
131
132

## It’s just JavaScript!

133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
Given that a JSDF file is just JavaScript, and includes a [UMD](https://github.com/umdjs/umd)-like declaration in its header (the first two lines), you can simply `require()` it as a module in Node.js or even load it in a script tag.

For example, create an _index.html_ file with the following content in the same folder as the other script and serve it locally using [Site.js](https://sitejs.org) and you will see the data printed out in your browser:

```html
<script src="db/people.js"></script>
<h1>People</h1>
<ul>
<script>
  people.forEach(person => {
    document.write(`<li>${person.name} (${person.age} years old)</li>`)
  })
</script>
</ul>
```

149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
## Supported and unsupported data types.

Just because it’s JavaScript, it doesn’t mean that you can throw anything into JSDB and expect it to work.

### Supported data types

  - `Number`
  - `Boolean`
  - `String`
  - `Object`
  - `Date`
  - `Symbol`
  - [Custom data types](#custom-data-types) (see below).

### Custom data types

Custom data types (instances of your own classes) are also supported.

During serialisation, class information for custom data types will be persisted.

During deserialisation, if the class in question exists in memory, your object will be correctly initialised as an instance of that class. If the class does not exist in memory, your object will be initialised as a plain JavaScript object.

### Unsupported data types

If you try to add an instance of an unsupported data type to a JSDB table, you will get a `TypeError`.

The following data types are currently unsupported but support is planned for the future:

  - `Map` (and `WeakMap`)
  - `Set` (and `WeakSet`)
  - Binary collections (`ArrayBuffer`, `Float32Array`, `Float64Array`, `Int8Array`, `Int16Array`, `Int32Array`, `TypedArray`, `Uint8Array`, `Uint16Array`, `Uint32Array`, and `Uint8ClampedArray`)

The following intrinsic objects are not supported as they don’t make sense to support:

  - Intrinsic objects (`DataView`, `Function`, `Generator`, `Promise`, `Proxy`, `RegExp`)
  - Error types (`Error`, `EvalError`, `RangeError`, `ReferenceError`, `SyntaxError`, `TypeError`, and `URIError`)

186
187
188
189
190
191
192
193
194
195
196
197
198
## Important security note

Note that JSDF is __not__ a data exchange format. Since it contains JavaScript code that is run, you must only load JSDF files from a domain that you own and control and have a secure connection to.

__Do not load in JSDF files from third parties.__

If you want a data _exchange_ format, use [JSON](https://www.json.org/json-en.html).

Remember:

  - JSON is a terrible format for a database but a great format for data exchange.
  - JSDF is a terrible format for data exchange but a great format for a JavaScript database.

199
200
## JavaScript Query Language (JSQL)

201
202
203
Of course, when you load the data in directly, you are not running it inside JSDB so you cannot update the data or use the JavaScript Query Language (JSQL) to query it.

To test that out, open a Node.js command-line interface (run `node`) from the directory that your scripts are in and enter the following commands:
204
205

```js
Aral Balkan's avatar
Aral Balkan committed
206
const JSDB = require('@small-tech/jsdb')
207
208

// This will load test database with the people table we created earlier.
209
const db = JSDB.open('db')
210

211
212
// Let’s carry out a query that should find us Osky.
console.log(db.people.where('age').isLessThan(21).get())
213
```
214

Aral Balkan's avatar
Aral Balkan committed
215
216
For details, see the [JSQL Reference](#jsql-reference) section.

217

218
219
## Compaction

220
221
___Note:__ I’m currently reviewing how compacting works. In a server setting, it __is__ important how fast the server restarts so I plan to make compaction a low-CPU background process that runs on a given interval instead of at every startup. This may have to be a post version 1.0 refactor._

222
223
When you load in a JSDB table, by default JSDB will compact the JSDF file.

224
Compaction is important for two reasons; during compaction:
225

226
227
  - Deleted data is actually deleted from disk. (Privacy.)
  - Old versions of updated data are actually removed. (Again, privacy.)
228
229
230

Compaction will also reduce the size of your tables.

231
That said, compaction is a relatively slow process that gets uniformly slower as the size of your database grows (it has O(N) time complexity as the whole database is recreated).
232

233
234
235
236
237
238
239
240
241
242
243
244
You do have the option to override the default behaviour and keep all history. You might want to do this, for example, if you’re creating a web app that lets you create a drawing and you want to play the drawing back stroke by stroke, etc.

Now that you’ve loaded the file back, look at the `./db/people.js` JSDF file again to see how it looks after compaction:

```js
globalThis._ = [];
(function () { if (typeof define === 'function' && define.amd) { define([], globalThis._); } else if (typeof module === 'object' && module.exports) { module.exports = globalThis._ } else { globalThis.people = globalThis._ } })();
_[0] = JSON.parse(`{"name":"Aral","age":43}`);
_[1] = JSON.parse(`{"name":"Laura","age":33}`);
_[2] = JSON.parse(`{"name":"Osky","age":8}`);
```

245
Ah, that is neater. You can see that Laura’s record is created with the correct age from the outset and Oskar’s name is set to its final value of Osky from the outset.
246

247
(You can find these examples in the `examples/basic` folder of the source code.)
Aral Balkan's avatar
Aral Balkan committed
248

Aral Balkan's avatar
Aral Balkan committed
249

250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
## Closing a database

Your database tables will be automatically closed if you exit your script. However, there might be times when you want to manually close a database (for example, to reopen it with different settings, etc.) In that case, you can call the asynchronous `close()` method on the database proxy.

Here’s what you’d do to close the database in the above example:

```js
async main () {
  // … 🠑 the earlier code from the example, above.

  await db.close()

  // The database and all of its tables are now closed.
  // It is now safe (and allowed) to reopen it.
}

main()
```

269
270
## Working with JSON

Aral Balkan's avatar
Aral Balkan committed
271
As mentioned earlier, JSDB writes out its tables as append-only logs of JavaScript statements in what we call JavaScript Data Format (JSDF). This is not the same as [JavaScript Object Notation (JSON)](https://www.json.org/json-en.html).
272

Aral Balkan's avatar
Aral Balkan committed
273
JSON is not a good format for a database but it is excellent – not to mention ubiquitous – for its original use case of data exchange. You can easily find or export datasets in JSON format. And using them in JSDB is effortless. Here’s an example that you can find in the `examples/json` folder of the source code:
274
275
276
277
278
279
280
281
282
283
284
285
286
287

Given a JSON data file of spoken languages by country in the following format:

```json
[
  {
    "country": "Aruba",
    "languages": [
      "Dutch",
      "English",
      "Papiamento",
      "Spanish"
    ]
  },
288
289
290
  {
    "etc.": "…"
  }
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
]
```

The following code will load in the file, populate a JSDB table with it, and perform a query on it:

```js
const fs = require('fs')
const JSDB = require('@small-tech/jsdb')

const db = JSDB.open('db')

// If the data has not been populated yet, populate it.
if (!db.countries) {
  const countries = JSON.parse(fs.readFileSync('./countries.json', 'utf-8'))
  db.countries = countries
}

// Query the data.
const countriesThatSpeakKurdish = db.countries.where('languages').includes('Kurdish').get()

console.log(countriesThatSpeakKurdish)
```

When you run it, you should see the following result:

```js
[
  {
    country: 'Iran',
    languages: [
      'Arabic',    'Azerbaijani',
      'Bakhtyari', 'Balochi',
      'Gilaki',    'Kurdish',
      'Luri',      'Mazandarani',
      'Persian',   'Turkmenian'
    ]
  },
  {
    country: 'Iraq',
    languages: [ 'Arabic', 'Assyrian', 'Azerbaijani', 'Kurdish', 'Persian' ]
  },
  { country: 'Syria', languages: [ 'Arabic', 'Kurdish' ] },
  { country: 'Turkey', languages: [ 'Arabic', 'Kurdish', 'Turkish' ] }
]
```

337

Aral Balkan's avatar
Aral Balkan committed
338
## Dispelling the magic and a pointing out a couple of gotchas
Aral Balkan's avatar
Aral Balkan committed
339

340
Here are a couple of facts to dispel the magic behind what’s going on:
Aral Balkan's avatar
Aral Balkan committed
341

342
343
344
345
  - What we call a _database_ in JSDB is just a regular directory on your file system.
  - Inside that directory, you can have zero or more tables.
  - A table is a JSDF file.
  - A JSDF file is a sequence of JavaScript statements that creates a data object (either an object or an array). It is an append-only log that is compacted at load. JSDF files are valid JavaScript files and should run correctly under any JavaScript interpreter.
346
347
  - When you open a database, you get a Proxy instance back, not an instance of JSDB.
  - Similarly, when you reference a table or the data within it, you are referencing proxy objects, not the table instance or the data itself.
Aral Balkan's avatar
Aral Balkan committed
348

Aral Balkan's avatar
Aral Balkan committed
349
350
### How the sausage is made

351
When you open a database, JSDB loads in any `.js` files it can find in your database directory. Doing so creates the data structures defined in those files in memory. Alongside, JSDB also creates a structure of [proxies](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Proxy) that mirrors the data structure and traps (captures) calls to get, set, or delete values. Every time you set or delete a value, the corresponding JavaScript statement is appended to your table on disk.
Aral Balkan's avatar
Aral Balkan committed
352

353
By calling the `where()` or `whereIsTrue()` methods, you start a [query](#jsql-reference). Queries help you search for specific bits of data. They are implemented using the get traps in the proxy.
Aral Balkan's avatar
Aral Balkan committed
354

Aral Balkan's avatar
Aral Balkan committed
355
356
### Gotchas and limitations

357
Given that a core goal for JSDB is to be transparent, you will mostly feel like you’re working with regular JavaScript collections (objects and arrays) instead of a database. That said, there are a couple of gotchas and limitations that arise from the use of proxies and the impedance mismatch between synchronous data manipulation in JavaScript and the asynchronous nature of file handling:
Aral Balkan's avatar
Aral Balkan committed
358

359
  1. __You can only have one copy of a database open at one time.__ Given that tables are append-only logs, having multiple streams writing to them would corrupt your tables. The JSDB class enforces this by forcing you to use the `open()` factory method to create or load in your databases.
Aral Balkan's avatar
Aral Balkan committed
360

361
  2. __You cannot reassign a value to your tables without first deleting them.__ Since assignment is a synchronous action and since we cannot safely replace the existing table on disk with a different one synchronously, you must first call the asynchronous `delete()` method on a table instance before assigning a new value for it on the database, thereby creating a new table.
Aral Balkan's avatar
Aral Balkan committed
362

363
364
365
      ```js
      async main () {
        // … 🠑 the earlier code from the example, above.
Aral Balkan's avatar
Aral Balkan committed
366

367
        await db.people.delete()
Aral Balkan's avatar
Aral Balkan committed
368

369
        // The people table is now deleted and we can recreate it.
Aral Balkan's avatar
Aral Balkan committed
370

371
372
373
374
        // This is OK.
        db.people = [
          {name: 'Ed Snowden', age: 37}
        ]
Aral Balkan's avatar
Aral Balkan committed
375

376
377
378
379
380
381
382
383
384
        // This is NOT.
        try {
          db.people = [
            {name: 'Someone else', age: 100}
          ]
        } catch (error) {
          console.log('This throws as we haven’t deleted the table first.')
        }
      }
385

386
387
      main()
      ```
388

389
390
391
392
393
394
395
396
  3. __There are certain reserved words you cannot use in your data.__ This is a trade-off between usability and polluting the mirrored proxy structure. JSDB strives to keep reserved words to a minimum.

        This is the full list:

        |                            | Reserved words                                                                 |
        | -------------------------- | ------------------------------------------------------------------------------ |
        | __As table name__          | `close`                                                                        |
        | __Property names in data__ | `where`, `whereIsTrue`, `addListener`, `removeListener`, `delete`, `__table__` |
397

398
        Note: You can use the `__table__` property from any level of your data to get a reference to the table instance (`JSTable` instance) that it belongs to. This is mostly for internal use but it’s there if you need it.
399

400
401
### Table events

402
403
You can listen for the following events on tables:

404
405
| Event name | Description                           |
| ---------- | ------------------------------------- |
406
| persist    | The table has been persisted to disk. |
407
408
| delete     | The table has been deleted from disk. |

409
410
411
412
413
414
415
416
417
418
#### Example

The following handler will get called whenever a change is persisted to disk for the `people` table:

```js
db.people.addListener('persist', (table, change) => {
  console.log(`Table ${table.tableName} persisted change ${change.replace('\n', '')} to disk.`)
})
```

419

Aral Balkan's avatar
Aral Balkan committed
420
421
## JSQL Reference

Aral Balkan's avatar
Aral Balkan committed
422
The examples in the reference all use the following random dataset. _Note, I know nothing about cars, the tags are also arbitrary. Don’t @ me ;)_
Aral Balkan's avatar
Aral Balkan committed
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484

```js
const cars = [
  { make: "Subaru", model: "Loyale", year: 1991, colour: "Fuscia", tags: ['fun', 'sporty'] },
  { make: "Chevrolet", model: "Suburban 1500", year: 2004, colour: "Turquoise", tags: ['regal', 'expensive'] },
  { make: "Honda", model: "Element", year: 2004, colour: "Orange", tags: ['fun', 'affordable'] },
  { make: "Subaru", model: "Impreza", year: 2011, colour: "Crimson", tags: ['sporty', 'expensive']},
  { make: "Hyundai", model: "Santa Fe", year: 2009, colour: "Turquoise", tags: ['sensible', 'affordable'] },
  { make: "Toyota", model: "Avalon", year: 2005, colour: "Khaki", tags: ['fun', 'affordable']},
  { make: "Mercedes-Benz", model: "600SEL", year: 1992, colour: "Crimson", tags: ['regal', 'expensive', 'fun']},
  { make: "Jaguar", model: "XJ Series", year: 2004, colour: "Red", tags: ['fun', 'expensive', 'sporty']},
  { make: "Isuzu", model: "Hombre Space", year: 2000, colour: "Yellow", tags: ['sporty']},
  { make: "Lexus", model: "LX", year: 1997, colour: "Indigo", tags: ['regal', 'expensive', 'AMAZING'] }
]
```

### Starting a query (the `where()` method)

```js
const carsMadeIn1991 = db.cars.where('year').is(1991).get()
```

The `where()` method starts a query.

You call it on a table reference. It takes a property name (string) as its only argument and returns a query instance.

On the returned query instance, you can call various operators like `is()` or `startsWith()`.

Finally, to invoke the query you use one one of the invocation methods: `get()`, `getFirst()`, or `getLast()`.

### The anatomy of a query.

Idiomatically, we chain the operator and invocation calls to the `where` call and write our queries out in a single line as shown above. However, you can split the three parts up, should you so wish. Here’s such an example, for academic purposes.

This starts the query and returns an incomplete query object:

```js
const incompleteCarYearQuery = db.cars.where('year')
```

Once you call an operator on a query, it is considered complete:

```js
const completeCarYearQuery = incompleteCarYearQuery.is(1991)
```

To execute a completed query, you can use one of the invocation methods: `get()`, `getFirst()`, or `getLast()`.

Note that `get()` returns an array of results (which might be an empty array) while `getFirst()` and `getLast()` return a single result (which may be `undefined`).

```js
const resultOfCarYearQuery = completeCarYearQuery.get()
```

Here are the three parts of a query shown together:

```js
const incompleteCarYearQuery = db.cars.where('year')
const completeCarYearQuery = incompleteCarYearQuery.is(1991)
const resultOfCarYearQuery = completeCarYearQuery.get()
```

485
Again, idiomatically, we chain the operator and invocation calls to the `where()` call and write our queries out in a single line like this:
Aral Balkan's avatar
Aral Balkan committed
486
487
488
489
490

```js
const carsMadeIn1991 = db.cars.where('year').is(1991).get()
```

491
### Connectives (`and()` and `or()`)
Aral Balkan's avatar
Aral Balkan committed
492

493
You can chain conditions onto a query using the connectives `and()` and `or()`. Using a connective transforms a completed query back into an incomplete query awaiting an operator. e.g.,
Aral Balkan's avatar
Aral Balkan committed
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546

```js
const veryOldOrOrangeCars = db.cars.where('year').isLessThan(2000).or('colour').is('Orange').get()
```

#### Example

```js
const carsThatAreFunAndSporty = db.cars.where('tags').includes('fun').and('tags').includes('sporty').get()
```

#### Result

```js
[
  { make: "Subaru", model: "Loyale", year: 1991, colour: "Fuscia", tags: ['fun', 'sporty'] },
  { make: "Jaguar", model: "XJ Series", year: 2004, colour: "Red", tags: ['fun', 'expensive', 'sporty']},
]
```

### Custom queries (`whereIsTrue()`)

For more complex queries – for example, if you need to include parenthetical grouping – you can compose your JSQL by hand. To do so, you call the `whereIsTrue()` method on a table instead of the `where()` method and you pass it a full JSQL query string. A completed query is returned.

When writing your custom JSQL query, prefix property names with `valueOf.`.

#### Example

```js
const customQueryResult = db.cars.whereIsTrue(`(valueOf.tags.includes('fun') && valueOf.tags.includes('affordable')) || (valueOf.tags.includes('regal') && valueOf.tags.includes('expensive'))`).get()
```

#### Result

```js
[
  { make: 'Chevrolet', model: 'Suburban 1500', year: 2004, colour: 'Turquoise', tags: [ 'regal', 'expensive' ] },
  { make: 'Honda', model: 'Element', year: 2004, colour: 'Orange', tags: [ 'fun', 'affordable' ] },
  { make: 'Toyota', model: 'Avalon', year: 2005, colour: 'Khaki', tags: [ 'fun', 'affordable' ] },
  { make: 'Mercedes-Benz', model: '600SEL', year: 1992, colour: 'Crimson', tags: [ 'regal', 'expensive', 'fun' ] },
  { make: 'Lexus', model: 'LX', year: 1997, colour: 'Indigo', tags: [ 'regal', 'expensive', 'AMAZING' ] }
]
```

### Relational operators

  - `is()`, `isEqualTo()`, `equals()`
  - `isNot()`, `doesNotEqual()`
  - `isGreaterThan()`
  - `isGreaterThanOrEqualTo()`
  - `isLessThan()`
  - `isLessThanOrEqualTo()`

547
Note: operators listed on the same line are aliases and may be used interchangeably (e.g., `isNot()` and `doesNotEqual()`).
Aral Balkan's avatar
Aral Balkan committed
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642

#### Example (is)

```js
const carWhereYearIs1991 = db.cars.where('year').is(1991).getFirst()
```

#### Result (is)

```js
{ make: "Subaru", model: "Loyale", year: 1991, colour: "Fuscia", tags: ['fun', 'sporty'] }
```

#### Example (isNot)

```js
const carsWhereYearIsNot1991 = db.cars.where('year').isNot(1991).get()
```

#### Result (isNot)

```js
[
  { make: "Chevrolet", model: "Suburban 1500", year: 2004, colour: "Turquoise", tags: ['regal', 'expensive'] },
  { make: "Honda", model: "Element", year: 2004, colour: "Orange", tags: ['fun', 'affordable'] },
  { make: "Subaru", model: "Impreza", year: 2011, colour: "Crimson", tags: ['sporty', 'expensive']},
  { make: "Hyundai", model: "Santa Fe", year: 2009, colour: "Turquoise", tags: ['sensible', 'affordable'] },
  { make: "Toyota", model: "Avalon", year: 2005, colour: "Khaki", tags: ['fun', 'affordable'] },
  { make: "Mercedes-Benz", model: "600SEL", year: 1992, colour: "Crimson", tags: ['regal', 'expensive', 'fun'] },
  { make: "Jaguar", model: "XJ Series", year: 2004, colour: "Red", tags: ['fun', 'expensive', 'sporty'] },
  { make: "Isuzu", model: "Hombre Space", year: 2000, colour: "Yellow", tags: ['sporty'] },
  { make: "Lexus", model: "LX", year: 1997, colour: "Indigo", tags: ['regal', 'expensive', 'AMAZING'] }
]
```

Note how `getFirst()` returns the first item (in this case, an _object_) whereas `get()` returns the whole _array_ of results.

The other relational operators work the same way and as expected.

### String subset comparison operators

  - `startsWith()`
  - `endsWith()`
  - `includes()`
  - `startsWithCaseInsensitive()`
  - `endsWithCaseInsensitive()`
  - `includesCaseInsensitive()`

The string subset comparison operators carry out case sensitive string subset comparisons. They also have case insensitive versions that you can use.

#### Example (`includes()` and `includesCaseInsensitive()`)

```js
const result1 = db.cars.where('make').includes('su').get()
const result2 = db.cars.where('make').includes('SU').get()
const result3 = db.cars.where('make').includesCaseInsensitive('SU')
```

#### Result 1

```js
[
  { make: "Isuzu", model: "Hombre Space", year: 2000, colour: "Yellow", tags: ['sporty']}
]
```

Since `includes()` is case sensitive, the string `'su`' matches only the make `Isuzu`.

#### Result 2

```js
[]
```

Again, since `includes()` is case sensitive, the string `'SU`' doesn’t match the make of any of the entries.

#### Result 3

```js
[
  { make: "Subaru", model: "Impreza", year: 2011, colour: "Crimson", tags: ['sporty', 'expensive'] },
  { make: "Isuzu", model: "Hombre Space", year: 2000, colour: "Yellow", tags: ['sporty'] }
]
```

Here, `includesCaseInsensitive('SU')` matches both the `Subaru` and `Isuzu` makes due to the case-insensitive string comparison.

### Array inclusion check operator

  - `includes()`

The `includes()` array inclusion check operator can also be used to check for the existence of an object (or scalar value) in an array.

Note that the `includesCaseInsensitive()` string operator cannot be used for this purpose and will throw an error if you try.

Aral Balkan's avatar
Aral Balkan committed
643
#### Example (`includes()` array inclusion check):
Aral Balkan's avatar
Aral Balkan committed
644
645
646
647
648

```js
const carsThatAreRegal = db.cars.where('tags').includes('regal').get()
```

Aral Balkan's avatar
Aral Balkan committed
649
#### Result (`includes()` array inclusion check)
Aral Balkan's avatar
Aral Balkan committed
650
651
652
653
654
655
656
657
658

```js
[
  { make: "Chevrolet", model: "Suburban 1500", year: 2004, colour: "Turquoise", tags: ['regal', 'expensive'] },
  { make: "Mercedes-Benz", model: "600SEL", year: 1992, colour: "Crimson", tags: ['regal', 'expensive', 'fun']},
  { make: "Lexus", model: "LX", year: 1997, colour: "Indigo", tags: ['regal', 'expensive', 'AMAZING'] }
]
```

Aral Balkan's avatar
Aral Balkan committed
659
660
## Performance characteristics

661
  - The time complexity of reads and writes are both O(1).
662
  - Reads are fast (take fraction of a millisecond and are about an order of magnitude slower than direct memory reads).
Aral Balkan's avatar
Aral Balkan committed
663
  - Writes are fast (in the order of a couple of milliseconds on tests on a dev machine).
Aral Balkan's avatar
Aral Balkan committed
664

665
## Limits
666

667
668
  - Your database size is limited by available memory.
  - If your database size is larger than > 1GB, you should start your node process with a larger heap size than the default (~1.4GB). E.g., to set aside 8GB of heap space:
Aral Balkan's avatar
Aral Balkan committed
669

670
671
672
  ```
  node --max-old-space-size=8192 why-is-my-database-so-large-i-hope-im-not-doing-anything-shady.js
  ```
Aral Balkan's avatar
Aral Balkan committed
673

674
675
## Memory Usage

676
The reason JSDB is fast is because it keeps the whole database in memory. Also, to provide a transparent persistence and query API, it maintains a parallel object structure of proxies. This means that the amount of memory used will be multiples of the size of your database on disk and exhibits O(N) memory complexity.
677

678
Initial load time and full table write/compaction both exhibit O(N) time complexity.
679

680
For example, here’s just one sample from a development laptop using the simple performance example in the `examples/performance` folder of the source code which creates random records that are around ~2KB in size each:
681

682
683
684
685
686
| Number of records | Table size on disk | Memory used | Initial load time | Full table write/compaction time |
| ----------------- | ------------------ | ----------- | ----------------- | -------------------------------- |
| 1,000             | 2.5MB              | 15.8MB      | 41.6ms            | 2.7 seconds                      |
| 10,000            | 25MB               | 121.4MB     | 380.2ms           | 26 seconds                       |
| 100,000           | 244MB              | 1.2GB       | 5.5 seconds       | 4.6 minutes                      |
687

688
(The baseline app used about 14.6MB without any table in memory. The memory used column subtracts that from the total reported memory so as not to skew the smaller dataset results.)
689

690
691
692
693
694
695
696
697
698
## Developing

Please open an issue before starting to work on pull requests.

1. Clone this repository.
2. `npm i`
3. `npm test`

For code coverage, run `npm run coverage`.
699

Aral Balkan's avatar
Aral Balkan committed
700
701
## Related projects, inspiration, etc.

702
  - [Initial brainstorming (query language)](https://gist.github.com/aral/fc4115fdf338e02d735ae58e245817ce)
Aral Balkan's avatar
Aral Balkan committed
703
704
705
706
  - [proxy-fun](https://github.com/mikaelbr/awesome-es2015-proxy)
  - [filejson](https://github.com/bchr02/filejson)
  - [Declaraoids](https://github.com/Matsemann/Declaraoids/blob/master/src/declaraoids.js)
  - [ScunMEngine](https://github.com/jlvaquero/SCUNM/blob/master/SCUNMEngine/SCUNMEngine.js)
Aral Balkan's avatar
Aral Balkan committed
707
708
709
710
711
712
713
714
715

## Like this? Fund us!

[Small Technology Foundation](https://small-tech.org) is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share [our vision](https://small-tech.org/about/#small-technology) and want to support our work, please [become a patron or donate to us](https://small-tech.org/fund-us) today and help us continue to exist.

## Copyright

716
&copy; 2020 [Aral Balkan](https://ar.al), [Small Technology Foundation](https://small-tech.org).