Implement X-Accel Redirect

This patch implements [X-Accel](https://www.nginx.com/resources/wiki/start/topics/examples/x-accel/)
redirect headers as an optional way for offloading static file delivery
from Express to Nginx, which is far better optimized for static file
delivery.

This provides a really easy to configure way for getting a huge
performance boost over delivering all files through Audiobookshelf.

How it works
------------

The way this works is basically that Audiobookshelf gets an HTTP request
for delivering a static file (let's say an audiobook). It will first
check the user is authorized and then convert the API path to a local
file path.

Now, instead of reading and delivering the file, Audiobookshelf will
return just the HTTP header with an additional `X-Accel-Redirect`
pointing to the file location on the file syste.

This header is picked up by Nginx which will then deliver the file.

Configuration
-------------

The configuration for this is very simple. You need to run Nginx as
reverse proxy and it must have access to your Audiobookshelf data
folder.

You then configure Audiobookshelf to use X-Accel by setting
`USE_X_ACCEL=/protected`. The path is the internal redirect path used by
Nginx.

In the Nginx configuration you then configure this location and map it
to the storage area to serve like this:

```
location /protected/ {
  internal;
  alias /;
}
```

That's all.

Impact
------

I just did a very simple performance test, downloading a 1170620819
bytes large audiobook file from another machine on the same network
like this, using `time -p` to measure how log the process took:

```sh
URL='https://url to audiobook…'

for i in `seq 1 50`
do
  echo "$i"
  curl -s -o /dev/null "${URL}"
done
```

This sequential test with 50 iterations and without x-accel resulted in:

```
real 413.42
user 197.11
sys 82.04
```

That is an average download speed of about 1080 MBit/s.

With X-Accel enabled, serving the files through Nginx, the same test
yielded the following results:

```
real 200.37
user 86.95
sys 29.79
```

That is an average download speed of about 2229 MBit/s, more than
doubling the previous speed.

I have also run the same test with 4 parallel processes and 25 downloads
each. Without x-accel, that test resulted in:

```
real 364.89
user 273.09
sys 112.75
```

That is an average speed of about 2448 MBit/s.

With X-Accel enabled, the parallel test also shows a significant
speedup:

```
real 167.19
user 195.62
sys 78.61
```

That is an average speed of about 5342 MBit/s.

While doing that, I also peaked at the system load which was a bit lower
when using X-Accel. Even though the system was delivering far more data.
But I just looked at the `load1` values and did not build a proper test
for that. That means, I cant provide any definitive data.

Supported Media
---------------

The current implementation works for audio files and book covers. There
are other media files which would benefit from this mechanism like feed
covers or author pictures.

But that's something for a future developer ;-)
This commit is contained in:
Lars Kiesow 2022-11-25 23:41:35 +01:00
parent da2d1455d7
commit 08250e266e
No known key found for this signature in database
GPG Key ID: 5DAFE8D9C823CE73
4 changed files with 26 additions and 5 deletions

View File

@ -48,6 +48,7 @@ class Server {
global.ConfigPath = fileUtils.filePathToPOSIX(Path.normalize(CONFIG_PATH))
global.MetadataPath = fileUtils.filePathToPOSIX(Path.normalize(METADATA_PATH))
global.RouterBasePath = ROUTER_BASE_PATH
global.XAccel = process.env.USE_X_ACCEL
if (!fs.pathExistsSync(global.ConfigPath)) {
fs.mkdirSync(global.ConfigPath)

View File

@ -201,6 +201,10 @@ class LibraryItemController {
return res.sendStatus(404)
}
if (global.XAccel) {
Logger.debug(`Use X-Accel to serve static file ${libraryItem.media.coverPath}`)
return res.status(204).header({'X-Accel-Redirect': global.XAccel + libraryItem.media.coverPath}).send()
}
return res.sendFile(libraryItem.media.coverPath)
}

View File

@ -51,6 +51,11 @@ class CacheManager {
// Cache exists
if (await fs.pathExists(path)) {
if (global.XAccel) {
Logger.debug(`Use X-Accel to serve static file ${path}`)
return res.status(204).header({'X-Accel-Redirect': global.XAccel + path}).send()
}
const r = fs.createReadStream(path)
const ps = new stream.PassThrough()
stream.pipeline(r, ps, (err) => {
@ -72,6 +77,11 @@ class CacheManager {
// Set owner and permissions of cache image
await filePerms.setDefault(path)
if (global.XAccel) {
Logger.debug(`Use X-Accel to serve static file ${writtenFile}`)
return res.status(204).header({'X-Accel-Redirect': global.XAccel + writtenFile}).send()
}
var readStream = fs.createReadStream(writtenFile)
readStream.pipe(res)
}

View File

@ -1,5 +1,6 @@
const express = require('express')
const Path = require('path')
const Logger = require('../Logger')
const { getAudioMimeTypeFromExtname } = require('../utils/fileUtils')
class StaticRouter {
@ -13,13 +14,18 @@ class StaticRouter {
init() {
// Library Item static file routes
this.router.get('/item/:id/*', (req, res) => {
var item = this.db.libraryItems.find(ab => ab.id === req.params.id)
const item = this.db.libraryItems.find(ab => ab.id === req.params.id)
if (!item) return res.status(404).send('Item not found with id ' + req.params.id)
var remainingPath = req.params['0']
var fullPath = null
if (item.isFile) fullPath = item.path
else fullPath = Path.join(item.path, remainingPath)
const remainingPath = req.params['0']
const fullPath = item.isFile ? item.path : Path.join(item.path, remainingPath)
// Allow reverse proxy to serve files directly
// See: https://www.nginx.com/resources/wiki/start/topics/examples/x-accel/
if (global.XAccel) {
Logger.debug(`Use X-Accel to serve static file ${fullPath}`)
return res.status(204).header({'X-Accel-Redirect': global.XAccel + fullPath}).send()
}
var opts = {}