[Design Pattern] Upload big file - 2. How to coordinate frontend and backend

How to coordinate frontend and backend?

File upload involves interaction between the frontend and backend, requiring the establishment of a standard communication protocol to accomplish the following core interactions:

1. File protocol creation 
2. Hash verification
3. Chunked data upload
4. Chunk merging

 

File protocol creation 

When the client sends a chunk to the server, it needs to inform the server which file upload the chunk belongs to. Therefore, a unique identifier is required to distinguish a specific file upload.

The file creation protocol is designed to generate and retrieve this unique identifier for the file upload process.

From Client side, it sends following information to BFF

// GET request
Upload-File-Size
Upload-File-Name
Upload-File-MIME

BFF need to tell Frontend:

status
uploadToken
chunkSize

• uploadToken: A unique identifier for the file upload process. Can be generated by uuid
• chunksize: The size of each chunk in bytes.

 

Hash verification

The client may need to verify the hash of a single chunk or the entire file. The server should provide the client with the current status of these hashes.

Frontend need to send:

Upload-Token
Upload-Hash-Type: chunk | file
Upload-Hash

BFF need to tell:

status
hasFile

• hasFile: information is important, if hasFile, then can skip that chunk upload

 

For the whole file, BFF also need to response:

rest: [[], []] // each inner array represent a chunk [from, end]
url

• upload-Hash-Type: Can take the value chunk or file, representing the hash of a chunk or the entire file, respectively.
• upload-Hash: The specific hash value of the chunk or file.
• hasFile: Indicates whether the server already stores the corresponding chunk or file.
• rest: A response field specific to file hash verification, indicating which hashes are still missing for the file.
• url: A response field specific to file hash verification; if the file upload is complete, this field indicates the file's request URL

 

 Chunked data upload

Upload specific file chunk data through this protocol.

About the Content-Type:

1. For General File Uploads

• multipart/form-data: Used for uploading files along with other form fields in a single request. This is the most common content type for file uploads in forms.
• application/octet-stream: Used for binary file uploads where the file is sent directly in the request body without any additional encoding.

2. For Specific File Types

• text/plain: For plain text files.
• application/json: If the file content is JSON or metadata is included as JSON.
• image/png, image/jpeg, image/gif: For image files.
• application/pdf: For PDF files.
• application/zip, application/x-tar: For compressed files.

3. For Chunked Uploads

• application/octet-stream: Commonly used for uploading binary chunks.
• Custom Content-Type: Some systems may define custom Content-Type values, such as application/vnd.upload-chunk or similar, depending on implementation.

 

 

Chunk merging

After all chunks have been uploaded, use this protocol to request the server to complete chunk merging.