Implement multipart parsing

[ronag]

Our extractBody method is missing support for parsing multipart/form-data responses.

[ronag]

I'm not sure who/why someone would send multipart/form-data from a server so would appreciate some requests on this. Doesn't seem to be a very useful feature.

[octet-stream]

Hey, it seems like node-fetch is going to have multipart parser soon, if this PR gets merged. Maybe this would be helpful for you somehow?

[jimmywarting]

I think it seems useful, i mean https://www.npmjs.com/package/form-data has 4.5M download / week, + it's a spec'ed thing

I have long been wanting to have a formData method in node-fetch

[MrWaip]

Hello. Telegram bot api accept only FormData for files
https://core.telegram.org/bots/api#inputfile

[blittle]

I'm not sure who/why someone would send multipart/form-data from a server so would appreciate some requests on this. Doesn't seem to be a very useful feature.

Correct me if I'm wrong, but isn't the bigger use case the server parsing multipart form data from the client? The example from the node-fetch PR iml is:

import { Response } from 'node-fetch'

http.createServer(async function (req, res) {
  const formData = await new Response(req, { headers: req.headers }).formData()
  const file = formData.get('avatar')
  
  var arrayBuffer = await file.arrayBuffer()
  var text = await file.text()
  var whatwgReadableStream = file.stream()
});

It would be great if undici had the same support for multipart form data.

[jimmywarting]

the example should actually be something like this:

http.createServer(async function (req, res) {
  const url = `${req.protocol}://${req.get('host')}${req.originalUrl}`
  const request new Request(url, { body: req, method: req.method, headers: req.headers })
  const formdata = request.formData()
  const file = formData.get('avatar')
  
  var arrayBuffer = await file.arrayBuffer()
  var text = await file.text()
  var whatwgReadableStream = file.stream()
});

But it's easier/quicker to construct a Response that don't need any url or extra options
the idea of having a formdata parsing and shipping the body.formData() comes from service workers
a service worker is meant to intercept the request and be able to parse the payload when needed and insert any missing fields like a authentication key or something

I'm not sure who/why someone would send multipart/form-data from a server so would appreciate some requests on this. Doesn't seem to be a very useful feature.

It's not about responding with a multipart/form-data response from a server like in: fetch(url).then(res => {res.formData()})
cuz again that would be weird if a server responded with a multipart/form-data. It's rather more about parsing incoming request to the http server which alot of folk use nodejs for.

I have long been wishing for a body.formData() but I never wanted to have that kind of functionality for getting such responses back from the server. it was b/c I wished for the servers IncommingRequest to mimic more like a service worker request and behaving more like the Request class... Where parsing IncommingRequest with body.formData() is essential so you wouldn't need busboy, formidable or any other library that you also need to learn how it works.

But again for a server to respond with a multipart/form-data isn't all that bad. It's a grate thing to use for sending multiple files back to the client now that fetch also has .formData(). And I have used it once in the web for transferering files as well.

// server.js
http.createServer(async function (req, res) {
  const formData = new FormData()
  fd.set('/uploads/logo.png', new File(...))
  fd.set('/uploads/movie-introduction.mp4', new File(...))
  const {body} = new Response(formData)
  stream.Readable.from(body).pipe(res)
})

// client.js
const res = await fetch('/get-files')
const fd = await res.formData()
const files = [...fd]
// use the files for something

This is much easier than bringing in a zip for packing and unpacking on both the server and on the client if you for any reason needs to get multipe files from the server. I have done this once before myself. But this is unherd of cuz never once have we had xhr.responseType = 'FormData' and not many are using service worker to intercept the request payload. this trully is a missing functionallity that we yet haven't taped into yet if there is such a thing to parse formdata in the other direction as well (from server to client).

---

I also added it cuz it's also part of the spec which might be the biggest reason to impl it so folk get what they expect

[ronag]

@jimmywarting: I wouldn't mind adding support for it. Would you be ok if someone port https://github.com/node-fetch/node-fetch/blob/main/src/utils/multipart-parser.js & https://github.com/node-fetch/node-fetch/pull/1314/files over to undici? @mcollina wdyt?

[jimmywarting]

Sure, Be my guest take what u want. The source is based on a quite old version of formidable that i patched quite a bit in 2017 (to get rid of some core nodejs parts like Buffer and Streams) so that it would make a friendlier browser version for github/fetch polyfill but never got merged. it was patched even more when folks reviewed my PR to node-fetch for some performences improvements

also take a look into busboy if their new parser is any faster. i haven't tested it against node-fetch new parser

I think the owner of busboy said that it dose some more extra stuff that don't co-here with the FormData parser spec like supporting non-UTF8 encodings

---

also have a other question... would this formData() parser make it into stream consumers as well somehow?

import {formData} from 'node:stream/consumers';
const fd = await formData(iterable, {boundary: req.boundary})
fd // is instancesof FormData

Think it would also be cool to have something more low level like

import {formData, blob} from 'node:stream/consumers'

// returns another asyncIterable (maybe the stream consumer name `formData` is the wrong name for it)
const asyncIterable = formData(req, {boundary: req.boundary})

for await (const [name, entry, meta] of asyncIterable) {
  if (typeof entry === 'string') {
    // it's normal regular string, should this also be streamable, what if the payload is really large?
  } else {
    // it's either a File class or something that is streamable/iterable as well.
    const blob = await blob(entry, { type: meta.contentType || meta.type }) // wish for type option to exist here (it's missing)
    new File([blob], meta.fileName, { type: blob.type })
  }
}

another solution could be to treat every entry the same and the meta only changes depending on the type of file vs normal field:

import {multipart, blob, text} from 'node:stream/consumers'

// returns another asyncIterable
const asyncIterable = multipart(req, {boundary: req.boundary})

for await (const [entry, meta] of asyncIterable) {
  const headers = meta.headers // Headers is a instance of fetch `Headers` class
  const type = headers.get('content-type')
  const name = meta.fieldName 

  // parses the filename and the utf8 filename variant as well out of content-disposition header
  const fileName = meta.fileName // undefined || string
  
  const item = await (fileName ? blob(entry) : string(entry))
}

think i like this☝️ solution more then the previous 2 examples

[blittle]

We'd like to have multipart support for Shopify Hydrogen, where our API route handlers are just functions that take undici Request objects. See Shopify/hydrogen#450

[KhafraDev]

I have a working parser that just needs tests written before a PR (no code used from other parsers which should help with license issues), although tests keep stalling tap (same issue as client-fetch.js which causes the current failures in the CI).

const server = createServer(async (req, res) => {
    const chunks = []
    for await (const chunk of req) chunks.push(chunk)

    const resp = new Response(Buffer.concat(chunks), { headers: req.headers })
    const form = await resp.formData()
    
    const same = form.get('c')
    const hope = readFileSync('./index.html', 'utf-8')

    assert(same.name, 'index.html')
    assert(hope === await same.text())
    assert(form.get('a') === 'b')
    
    res.end('lol')
    server.close()
}).listen(0)

const form = new FormData()
form.set('a', 'b')
form.set('c', new Blob([readFileSync('./index.html')]), 'index.html')

// send POST req with form body here...

I'm just worried as it's less than 100 lines that there may be edge cases I've missed, although I've hopefully made it fool-proof 😅.