How do I send a buffer in an HTTP request?

Question

I have a file in memory (buffer) - there is no file on the file system. I want to send that buffer to another server that talks HTTP.

For example, some API A creates a file in memory, SignServer manipulates such files, and responds with a new buffer. My API takes the file from A and feeds it to SignServer.

I tried sending the file to SignServer in multiple ways, but it keeps responding with status 400 (missing field 'data' in request).

What I tried:

var http = require('http');
var querystring = require('querystring');

var data = querystring.stringify({
    workerName: 'PDFSigner',
    data: file_buffer
});

var request = new http.ClientRequest({
    hostname: 'localhost',
    port: 8080,
    path: '/signserver/process',
    method: 'GET',
    headers: {
        'Content-Type': 'application/x-www-form-urlencoded',
        // I also tried 'multipart/form-data'
        'Content-Length': Buffer.byteLength(data)
    }
});

request.end(data);

I tried printing data, and it showed:

workerName=PDFSigner&data=

Which is bad because data wasn't set to file_buffer. I tried printing file_buffer, and it does have content (not null, not undefined, actually has bytes inside). So stringifying the buffer gave an empty string.

I tried doing the same thing with the request module and it didn't work either.

Note that SignServer isn't written in Node nor JavaScript. It's a Java application, so it probably doesn't work with json (which is why I'm trying to do it with querystring). Yes, I tried sending json.

Please check this : https://nodejs.org/api/http.html#http_request_end_data_encoding_callback — Saulius Next, May 11 '17 at 09:09
@SauliusNext - Ok, so that's how I send binary data. But I need to send two things - one is a string, the other is binary, and I need to give them names so that the API can tell them apart. — Ivan Rubinson, May 11 '17 at 09:14
Interesting read: https://github.com/nodejs/node-v0.x-archive/issues/5827 — Ivan Rubinson, May 11 '17 at 09:17
Another interesting read: https://github.com/tj/node-querystring/issues/45 — Ivan Rubinson, May 11 '17 at 09:20
@robertklep Can I send binary data (buffer) not in a `multipart/form-data`? — Ivan Rubinson, May 11 '17 at 09:21
@IvanRubinson my guess is that the server you're trying to contact deals with file uploads? If so, it will be assuming `multipart/form-data`. Files typically aren't sent using `application/x-www-form-urlencoded` because it's not really suitable for large(ish) binary data. — robertklep, May 11 '17 at 09:23
@robertklep Here's a new question about how to do just that: http://stackoverflow.com/questions/43913650/how-to-send-a-buffer-in-form-data-to-signserver — Ivan Rubinson, May 11 '17 at 11:19

score 4 · Accepted Answer · answered May 11 '17 at 09:31

The reason why data is set to an empty string is described in this issue and the solution is given in this issue.

escape and stringify the buffer like so:

var data = querystring.stringify({
    workerName: 'PDFSigner',
    data: escape(file_buffer).toString('binary')
});

As @robertklep mentioned, your other problem is that you can't send a big file using application/x-www-form-urlencoded. You'd need to do it with multipart/form-data.

How do I send a buffer in an HTTP request?

1 Answers1

Linked