【SpringBoot整合系列】SpringBoot 实现大文件分片上传、断点续传及秒传

SpringBoot实现大文件上传【分片上传、断点续传、秒传】

苏生Susheng

3065人浏览 · 2024-04-09 13:16:31

苏生Susheng · 2024-04-09 13:16:31 发布

功能介绍

文件上传

小文件（图片、文档、视频）上传可以直接使用很多ui框架封装的上传组件，或者自己写一个input 上传，利用FormData 对象提交文件数据，后端使用spring提供的MultipartFile进行文件的接收，然后写入即可。
但是对于比较大的文件，比如上传2G左右的文件（http上传），就需要将文件分片上传（file.slice()），否则中间http长时间连接可能会断掉

分片上传

分片上传，就是将所要上传的文件，按照一定的大小，将整个文件分隔成多个数据块（我们称之为Part）来进行分别上传，上传完之后再由服务端对所有上传的文件进行汇总整合成原始的文件

秒传

通俗的说，你把要上传的东西上传，服务器会先做MD5校验，如果服务器上有一样的东西，它就直接给你个新地址，其实你下载的都是服务器上的同一个文件
想要不秒传，其实只要让MD5改变，就是对文件本身做一下修改（改名字不行），例如一个文本文件，你多加几个字，MD5就变了，就不会秒传了

断点续传

断点续传是在下载或上传时，将下载或上传任务（一个文件或一个压缩包）人为的划分为几个部分，每一个部分采用一个线程进行上传或下载
如果碰到网络故障，可以从已经上传或下载的部分开始继续上传或者下载未完成的部分，而没有必要从头开始上传或者下载。本文的断点续传主要是针对断点上传场景。

大文件上传流程

前端对文件进行MD5加密，并且将文件按一定的规则分片
vue-simple-uploader先会发送get请求校验分片数据在服务端是否完整，如果完整则进行秒传，如果不完整或者无数据，则进行分片上传。
后台校验MD5值，根据上传的序号和分片大小计算相应的开始位置并写入该分片数据到文件中。

前端切片处理逻辑

在这里插入图片描述

后端处理切片的逻辑

在这里插入图片描述

流程解析

在created时，初始化uploader组件，指定分片大小、上传方式等配置。
在onFileAdded方法中，当选择文件计算MD5后，调用file.resume()开始上传。
file.resume()内部首先发送一个GET请求，询问服务端该文件已上传的分片。
服务端返回一个JSON，里面包含已上传分片的列表。
uploader组件调用checkChunkUploadedByResponse，校验当前分片是否在已上传的列表中。
对未上传的分片，file.resume()会继续触发上传该分片的POST请求。
POST请求会包含一个分片的数据和偏移量等信息。
服务端接收分片数据，写入文件的指定位置并返回成功响应。
uploader组件会记录该分片已上传完成。
依次上传完所有分片后，服务器端合并所有分片成一个完整的文件。
onFileSuccess被调用，通知上传成功。
这样通过GET请求询问已上传分片+POST上传未完成分片+校验的方式，实现了断点续传/分片上传。

后端代码实现

SpringBoot2.7.16+MySQL+JPA+hutool

功能目标

get请求接口校验上传文件MD5值和文件是否完整
post请求接收上传文件，并且计算分片，写入合成文件
文件完整上传完成时，往文件存储表tool_local_storage中加一条该文件的信息
get请求接口实现简单的文件下载

1.建表SQL

DROP TABLE IF EXISTS `file_chunk`;
CREATE TABLE `file_chunk`  (
`id` bigint(20) UNSIGNED NOT NULL AUTO_INCREMENT,
`file_name` varchar(255) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL COMMENT '文件名',
`chunk_number` int(11) NULL DEFAULT NULL COMMENT '当前分片，从1开始',
`chunk_size` float NULL DEFAULT NULL COMMENT '分片大小',
`current_chunk_size` float NULL DEFAULT NULL COMMENT '当前分片大小',
`total_size` double(20, 0) NULL DEFAULT NULL COMMENT '文件总大小',
`total_chunk` int(11) NULL DEFAULT NULL COMMENT '总分片数',
`identifier` varchar(45) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL COMMENT '文件标识',
`relative_path` varchar(255) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL COMMENT 'md5校验码',
`createtime` timestamp(0) NOT NULL DEFAULT CURRENT_TIMESTAMP,
`updatetime` timestamp(0) NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP(0),
PRIMARY KEY (`id`) USING BTREE
) ENGINE = InnoDB AUTO_INCREMENT = 1529 CHARACTER SET = utf8 COLLATE = utf8_general_ci ROW_FORMAT = Dynamic;
DROP TABLE IF EXISTS `tool_local_storage`;
CREATE TABLE `tool_local_storage`  (
`id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'ID',
`real_name` varchar(255) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL COMMENT '文件真实的名称',
`name` varchar(255) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL COMMENT '文件名',
`suffix` varchar(255) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL COMMENT '后缀',
`path` varchar(255) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL COMMENT '路径',
`type` varchar(255) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL COMMENT '类型',
`size` varchar(100) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL COMMENT '大小',
`identifier` varchar(255) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL COMMENT 'md5校验码\r\n',
`create_by` varchar(255) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL COMMENT '创建者',
`update_by` varchar(255) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL COMMENT '更新者',
`createtime` timestamp(0) NOT NULL DEFAULT CURRENT_TIMESTAMP,
`updatetime` timestamp(0) NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP(0),
PRIMARY KEY (`id`) USING BTREE
) ENGINE = InnoDB AUTO_INCREMENT = 3360 CHARACTER SET = utf8 COLLATE = utf8_general_ci COMMENT = '文件存储' ROW_FORMAT = Compact;

2.引入依赖

		<dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <version>5.1.47</version>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-data-jpa</artifactId>
        </dependency>
        <dependency>
            <groupId>cn.hutool</groupId>
            <artifactId>hutool-all</artifactId>
            <version>5.3.4</version>
        </dependency>
        <dependency>
            <groupId>commons-io</groupId>
            <artifactId>commons-io</artifactId>
            <version>2.6</version>
        </dependency>
        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
            <version>1.18.2</version>
        </dependency>
        <dependency>
            <groupId>com.alibaba</groupId>
            <artifactId>fastjson</artifactId>
            <version>1.2.70</version>
        </dependency>

3.实体类

package com.zjl.domin;

import com.fasterxml.jackson.annotation.JsonFormat;
import lombok.Data;
import org.springframework.web.multipart.MultipartFile;

import javax.persistence.*;
import java.io.Serializable;
import java.util.Date;

/**
 * @author: zjl
 * @datetime: 2024/4/9
 * @desc:
 */
@Data
@Entity
@Table(name = "file_chunk")
public class FileChunkParam implements Serializable {
    @Id
    @Column(name = "id")
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long id;

    @Column(name = "chunk_number")
    private Integer chunkNumber;

    @Column(name = "chunk_size")
    private Float chunkSize;

    @Column(name = "current_chunk_size")
    private Float currentChunkSize;

    @Column(name = "total_chunk")
    private Integer totalChunks;

    @Column(name = "total_size")
    private Double totalSize;

    @Column(name = "identifier")
    private String identifier;

    @Column(name = "file_name")
    private String filename;

    @Column(name = "relative_path")
    private String relativePath;

    @JsonFormat(shape = JsonFormat.Shape.STRING, pattern = "yyyy-MM-dd HH:mm:ss", timezone = "GMT+8")
    @Column(name = "createtime")
    private Date createtime;

    @JsonFormat(shape = JsonFormat.Shape.STRING, pattern = "yyyy-MM-dd HH:mm:ss", timezone = "GMT+8")
    @Column(name = "updatetime")
    private Date updatetime;

    @Transient
    private MultipartFile file;
}

package com.zjl.domin;

import cn.hutool.core.bean.BeanUtil;
import cn.hutool.core.bean.copier.CopyOptions;
import com.fasterxml.jackson.annotation.JsonFormat;
import lombok.Data;

import javax.persistence.*;
import java.io.Serializable;
import java.util.Date;

/**
 * @author: zjl
 * @datetime: 2024/4/9
 * @desc:
 */
@Data
@Entity
@Table(name = "tool_local_storage")
public class LocalStorage implements Serializable {

    @Id
    @Column(name = "id")
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long id;

    @Column(name = "real_name")
    private String realName;

    @Column(name = "name")
    private String name;

    @Column(name = "suffix")
    private String suffix;

    @Column(name = "path")
    private String path;

    @Column(name = "type")
    private String type;

    @Column(name = "size")
    private String size;

    @Column(name = "identifier")
    private String identifier;

    @JsonFormat(shape = JsonFormat.Shape.STRING, pattern = "yyyy-MM-dd HH:mm:ss", timezone = "GMT+8")
    @Column(name = "createtime")
    private Date createtime;

    @JsonFormat(shape = JsonFormat.Shape.STRING, pattern = "yyyy-MM-dd HH:mm:ss", timezone = "GMT+8")
    @Column(name = "updatetime")
    private Date updatetime;

    public LocalStorage() {
    }

    public LocalStorage(String realName, String name, String suffix, String path, String type, String size, String identifier) {
        this.realName = realName;
        this.name = name;
        this.suffix = suffix;
        this.path = path;
        this.type = type;
        this.size = size;
        this.identifier = identifier;
    }

    public LocalStorage(Long id, String realName, String name, String suffix, String path, String type, String size, String identifier) {
        this.id = id;
        this.realName = realName;
        this.name = name;
        this.suffix = suffix;
        this.path = path;
        this.type = type;
        this.size = size;
        this.identifier = identifier;
    }

    public void copy(LocalStorage source) {
        BeanUtil.copyProperties(source, this, CopyOptions.create().setIgnoreNullValue(true));
    }
}

4.响应模板

package com.zjl.domin;

import com.fasterxml.jackson.annotation.JsonInclude;
import lombok.Data;

/**
 * @author: zjl
 * @datetime: 2024/4/9
 * @desc:
 */
@Data
@JsonInclude(JsonInclude.Include.NON_EMPTY)
public class ResultVO<T> {
    /**
     * 错误码.
     */
    private Integer code;

    /**
     * 提示信息.
     */
    private String msg;

    /**
     * 具体内容.
     */
    private T data;

    public ResultVO(Integer code, String msg, T data) {
        this.code = code;
        this.msg = msg;
        this.data = data;
    }

    public ResultVO(Integer code, String msg) {
        this.code = code;
        this.msg = msg;
    }

    public ResultVO() {
    }
}

5.枚举类

package com.zjl.enums;

import lombok.Getter;

/**
 * @author: zjl
 * @datetime: 2024/4/9
 * @desc:
 */
public enum MessageEnum {
    /**
     * 消息枚举
     */
    FAIL(-1, "操作失败"),
    SUCCESS(200, "操作成功"),
    RECORD_NOT_EXISTED(1001, "记录不存在"),
    PARAM_NOT_NULL(1002, "参数不能为空"),
    PARAM_INVALID(1003, "参数错误"),
    UPLOAD_FILE_NOT_NULL(1004, "上传文件不能为空"),
    OVER_FILE_MAX_SIZE(1005, "超出文件大小");

    MessageEnum(int value, String text) {
        this.code = value;
        this.message = text;
    }

    @Getter
    private final int code;

    @Getter
    private final String message;

    public static MessageEnum valueOf(int value) {
        MessageEnum[] enums = values();
        for (MessageEnum enumItem : enums) {
            if (value == enumItem.getCode()) {
                return enumItem;
            }
        }
        return null;
    }
}

6.自定义异常

package com.zjl.exception;

import com.zjl.enums.MessageEnum;
import lombok.AllArgsConstructor;
import lombok.Getter;
import lombok.NoArgsConstructor;
import lombok.Setter;

/**
 * @author: zjl
 * @datetime: 2024/4/9
 * @desc:
 */
@Getter
@Setter
@NoArgsConstructor
@AllArgsConstructor
public abstract class BaseErrorException extends RuntimeException {

    private static final long serialVersionUID = 6386720492655133851L;
    private int code;
    private String error;

    public BaseErrorException(MessageEnum messageEnum) {
        this.code = messageEnum.getCode();
        this.error = messageEnum.getMessage();
    }
}

package com.zjl.exception;

import com.zjl.enums.MessageEnum;
import lombok.Data;

/**
 * @author: zjl
 * @datetime: 2024/4/9
 * @desc:
 */
@Data
public class BusinessException extends BaseErrorException {

    private static final long serialVersionUID = 2369773524406947262L;

    public BusinessException(MessageEnum messageEnum) {
        super(messageEnum);
    }

    public BusinessException(String error) {
        super.setCode(-1);
        super.setError(error);
    }
}

7.工具类

package com.zjl.utils;

import cn.hutool.core.io.IoUtil;
import cn.hutool.core.util.IdUtil;
import cn.hutool.poi.excel.BigExcelWriter;
import cn.hutool.poi.excel.ExcelUtil;
import com.zjl.enums.MessageEnum;
import com.zjl.exception.BusinessException;
import org.apache.commons.io.IOUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.web.multipart.MultipartFile;
import sun.misc.BASE64Encoder;

import javax.servlet.ServletOutputStream;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.*;
import java.net.URLEncoder;
import java.security.MessageDigest;
import java.text.DecimalFormat;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.List;
import java.util.Map;
/**
 * @author: zjl
 * @datetime: 2024/4/9
 * @desc:File工具类，扩展 hutool 工具包
 */
public class FileUtil extends cn.hutool.core.io.FileUtil {
    private static final Logger log = LoggerFactory.getLogger(FileUtil.class);
    /**
     * 系统临时目录
     * <br>
     * windows 包含路径分割符，但Linux 不包含,
     * 在windows \\==\ 前提下，
     * 为安全起见 同意拼装 路径分割符，
     * <pre>
     *       java.io.tmpdir
     *       windows : C:\Users/xxx\AppData\Local\Temp\
     *       linux: /temp
     * </pre>
     */
    public static final String SYS_TEM_DIR = System.getProperty("java.io.tmpdir") + File.separator;
    /**
     * 定义GB的计算常量
     */
    private static final int GB = 1024 * 1024 * 1024;
    /**
     * 定义MB的计算常量
     */
    private static final int MB = 1024 * 1024;
    /**
     * 定义KB的计算常量
     */
    private static final int KB = 1024;

    /**
     * 格式化小数
     */
    private static final DecimalFormat DF = new DecimalFormat("0.00");

    /**
     * MultipartFile转File
     */
    public static File toFile(MultipartFile multipartFile) {
        // 获取文件名
        String fileName = multipartFile.getOriginalFilename();
        // 获取文件后缀
        String prefix = "." + getExtensionName(fileName);
        File file = null;
        try {
            // 用uuid作为文件名，防止生成的临时文件重复
            file = File.createTempFile(IdUtil.simpleUUID(), prefix);
            // MultipartFile to File
            multipartFile.transferTo(file);
        } catch (IOException e) {
            log.error(e.getMessage(), e);
        }
        return file;
    }

    /**
     * 获取文件扩展名，不带 .
     */
    public static String getExtensionName(String filename) {
        if ((filename != null) && (filename.length() > 0)) {
            int dot = filename.lastIndexOf('.');
            if ((dot > -1) && (dot < (filename.length() - 1))) {
                return filename.substring(dot + 1);
            }
        }
        return filename;
    }

    /**
     * Java文件操作 获取不带扩展名的文件名
     */
    public static String getFileNameNoEx(String filename) {
        if ((filename != null) && (filename.length() > 0)) {
            int dot = filename.lastIndexOf('.');
            if ((dot > -1) && (dot < (filename.length()))) {
                return filename.substring(0, dot);
            }
        }
        return filename;
    }

    /**
     * 文件大小转换
     */
    public static String getSize(long size) {
        String resultSize;
        if (size / GB >= 1) {
            //如果当前Byte的值大于等于1GB
            resultSize = DF.format(size / (float) GB) + "GB   ";
        } else if (size / MB >= 1) {
            //如果当前Byte的值大于等于1MB
            resultSize = DF.format(size / (float) MB) + "MB   ";
        } else if (size / KB >= 1) {
            //如果当前Byte的值大于等于1KB
            resultSize = DF.format(size / (float) KB) + "KB   ";
        } else {
            resultSize = size + "B   ";
        }
        return resultSize;
    }

    /**
     * inputStream 转 File
     */
    static File inputStreamToFile(InputStream ins, String name) throws Exception {
        File file = new File(SYS_TEM_DIR + name);
        if (file.exists()) {
            return file;
        }
        OutputStream os = new FileOutputStream(file);
        int bytesRead;
        int len = 8192;
        byte[] buffer = new byte[len];
        while ((bytesRead = ins.read(buffer, 0, len)) != -1) {
            os.write(buffer, 0, bytesRead);
        }
        os.close();
        ins.close();
        return file;
    }

    /**
     * 将文件名解析成文件的上传路径
     */
    public static File upload(MultipartFile file, String filePath) {
        Date date = new Date();
        SimpleDateFormat format = new SimpleDateFormat("yyyyMMddhhmmssS");
        String name = getFileNameNoEx(file.getOriginalFilename());
        String suffix = getExtensionName(file.getOriginalFilename());
        String nowStr = "-" + format.format(date);
        try {
            String fileName = name + nowStr + "." + suffix;
            String path = filePath + fileName;
            // getCanonicalFile 可解析正确各种路径
            File dest = new File(path).getCanonicalFile();
            // 检测是否存在目录
            if (!dest.getParentFile().exists()) {
                if (!dest.getParentFile().mkdirs()) {
                    System.out.println("was not successful.");
                }
            }
            // 文件写入
            file.transferTo(dest);
            return dest;
        } catch (Exception e) {
            log.error(e.getMessage(), e);
        }
        return null;
    }

    /**
     * 导出excel
     */
    public static void downloadExcel(List<Map<String, Object>> list, HttpServletResponse response) throws IOException {
        String tempPath = SYS_TEM_DIR + IdUtil.fastSimpleUUID() + ".xlsx";
        File file = new File(tempPath);
        BigExcelWriter writer = ExcelUtil.getBigWriter(file);
        // 一次性写出内容，使用默认样式，强制输出标题
        writer.write(list, true);
        //response为HttpServletResponse对象
        response.setContentType("application/vnd.openxmlformats-officedocument.spreadsheetml.sheet;charset=utf-8");
        //test.xls是弹出下载对话框的文件名，不能为中文，中文请自行编码
        response.setHeader("Content-Disposition", "attachment;filename=file.xlsx");
        ServletOutputStream out = response.getOutputStream();
        // 终止后删除临时文件
        file.deleteOnExit();
        writer.flush(out, true);
        //此处记得关闭输出Servlet流
        IoUtil.close(out);
    }

    public static String getFileType(String type) {
        String documents = "txt pdf pps wps doc docx ppt pptx xls xlsx";
        String music = "mp3 wav wma mpa ram ra aac aif m4a";
        String video = "avi mpg mpe mpeg asf wmv mov qt rm mp4 flv m4v webm ogv ogg";
        String image = "bmp dib pcp dif wmf gif jpg tif eps psd cdr iff tga pcd mpt png jpeg";
        if (image.contains(type)) {
            return "图片";
        } else if (documents.contains(type)) {
            return "文档";
        } else if (music.contains(type)) {
            return "音乐";
        } else if (video.contains(type)) {
            return "视频";
        } else {
            return "其他";
        }
    }

    public static String getTransferFileType(String type) {
        String documents = "txt pdf pps wps doc docx ppt pptx xls xlsx";
        String music = "mp3 wav wma mpa ram ra aac aif m4a";
        String video = "avi mpg mpe mpeg asf wmv mov qt rm mp4 flv m4v webm ogv ogg";
        String image = "bmp dib pcp dif wmf gif jpg tif eps psd cdr iff tga pcd mpt png jpeg";
        if (image.contains(type)) {
            return "image";
        } else if (documents.contains(type)) {
            return "documents";
        } else if (music.contains(type)) {
            return "music";
        } else if (video.contains(type)) {
            return "video";
        } else {
            return "other";
        }
    }

    public static void checkSize(long maxSize, long size) {
        // 1M
        int len = 1024 * 1024;
        if (size > (maxSize * len)) {
            throw new BusinessException(MessageEnum.OVER_FILE_MAX_SIZE);
        }
    }

    /**
     * 判断两个文件是否相同
     */
    public static boolean check(File file1, File file2) {
        String img1Md5 = getMd5(file1);
        String img2Md5 = getMd5(file2);
        return img1Md5.equals(img2Md5);
    }

    /**
     * 判断两个文件是否相同
     */
    public static boolean check(String file1Md5, String file2Md5) {
        return file1Md5.equals(file2Md5);
    }

    private static byte[] getByte(File file) {
        // 得到文件长度
        byte[] b = new byte[(int) file.length()];
        try {
            InputStream in = new FileInputStream(file);
            try {
                System.out.println(in.read(b));
            } catch (IOException e) {
                log.error(e.getMessage(), e);
            }
        } catch (FileNotFoundException e) {
            log.error(e.getMessage(), e);
            return null;
        }
        return b;
    }

    private static String getMd5(byte[] bytes) {
        // 16进制字符
        char[] hexDigits = {'0', '1', '2', '3', '4', '5', '6', '7', '8', '9', 'a', 'b', 'c', 'd', 'e', 'f'};
        try {
            MessageDigest mdTemp = MessageDigest.getInstance("MD5");
            mdTemp.update(bytes);
            byte[] md = mdTemp.digest();
            int j = md.length;
            char[] str = new char[j * 2];
            int k = 0;
            // 移位 输出字符串
            for (byte byte0 : md) {
                str[k++] = hexDigits[byte0 >>> 4 & 0xf];
                str[k++] = hexDigits[byte0 & 0xf];
            }
            return new String(str);
        } catch (Exception e) {
            log.error(e.getMessage(), e);
        }
        return null;
    }

    /**
     * 下载文件
     *
     * @param request  /
     * @param response /
     * @param file     /
     */
    public static void downloadFile(HttpServletRequest request, HttpServletResponse response, File file, boolean deleteOnExit) throws UnsupportedEncodingException {
        response.setCharacterEncoding(request.getCharacterEncoding());
        response.setContentType("application/octet-stream");
        FileInputStream fis = null;

        String filename = filenameEncoding(file.getName(), request);
        try {
            fis = new FileInputStream(file);
            response.setHeader("Content-Disposition", String.format("attachment;filename=%s", filename));
            IOUtils.copy(fis, response.getOutputStream());
            response.flushBuffer();
        } catch (Exception e) {
            log.error(e.getMessage(), e);
        } finally {
            if (fis != null) {
                try {
                    fis.close();
                    if (deleteOnExit) {
                        file.deleteOnExit();
                    }
                } catch (IOException e) {
                    log.error(e.getMessage(), e);
                }
            }
        }
    }

    public static String getMd5(File file) {
        return getMd5(getByte(file));
    }

    public static String filenameEncoding(String filename, HttpServletRequest request) throws UnsupportedEncodingException {
        // 获得请求头中的User-Agent
        String agent = request.getHeader("User-Agent");
        // 根据不同的客户端进行不同的编码

        if (agent.contains("MSIE")) {
            // IE浏览器
            filename = URLEncoder.encode(filename, "utf-8");
        } else if (agent.contains("Firefox")) {
            // 火狐浏览器
            BASE64Encoder base64Encoder = new BASE64Encoder();
            filename = "=?utf-8?B?" + base64Encoder.encode(filename.getBytes("utf-8")) + "?=";
        } else {
            // 其它浏览器
            filename = URLEncoder.encode(filename, "utf-8");
        }
        return filename;
    }
}

8.Controller层

package com.zjl.controller;
import com.zjl.domin.FileChunkParam;
import com.zjl.domin.ResultVO;
import com.zjl.service.FileChunkService;
import com.zjl.service.FileService;
import com.zjl.service.LocalStorageService;
import lombok.extern.slf4j.Slf4j;
import org.springframework.web.bind.annotation.*;

import javax.annotation.Resource;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.IOException;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
/**
 * @author: zjl
 * @datetime: 2024/4/9
 * @desc:
 */
@RestController
@Slf4j
@RequestMapping("/api")
public class FileUploadController {

    @Resource
    private FileService fileService;
    @Resource
    private FileChunkService fileChunkService;
    @Resource
    private LocalStorageService localStorageService;

    @GetMapping("/upload")
    public ResultVO<Map<String, Object>> checkUpload(FileChunkParam param) {
        log.info("文件MD5:" + param.getIdentifier());
        List<FileChunkParam> list = fileChunkService.findByMd5(param.getIdentifier());
        Map<String, Object> data = new HashMap<>(1);
        // 判断文件存不存在
        if (list.size() == 0) {
            data.put("uploaded", false);
            return new ResultVO<>(200, "上传成功", data);
        }
        // 处理单文件
        if (list.get(0).getTotalChunks() == 1) {
            data.put("uploaded", true);
            data.put("url", "");
            return new ResultVO<Map<String, Object>>(200, "上传成功", data);
        }
        // 处理分片
        int[] uploadedFiles = new int[list.size()];
        int index = 0;
        for (FileChunkParam fileChunkItem : list) {
            uploadedFiles[index] = fileChunkItem.getChunkNumber();
            index++;
        }
        data.put("uploadedChunks", uploadedFiles);
        return new ResultVO<Map<String, Object>>(200, "上传成功", data);
    }

    @PostMapping("/upload")
    public ResultVO chunkUpload(FileChunkParam param) {
        log.info("上传文件：{}", param);
        boolean flag = fileService.uploadFile(param);
        if (!flag) {
            return new ResultVO(211, "上传失败");
        }
        return new ResultVO(200, "上传成功");
    }


    @GetMapping(value = "/download/{md5}/{name}")
    public void downloadbyname(HttpServletRequest request, HttpServletResponse response, @PathVariable String name, @PathVariable String md5) throws IOException {
        localStorageService.downloadByName(name, md5, request, response);
    }
}

9.FileService

package com.zjl.service;


import com.zjl.domin.FileChunkParam;

/**
 * @author: zjl
 * @datetime: 2024/4/9
 * @desc:
 */
public interface FileService {
    /**
     * 上传文件
     * @param param 参数
     * @return
     */
    boolean uploadFile(FileChunkParam param);
}

package com.zjl.service.impl;

import com.zjl.domin.FileChunkParam;
import com.zjl.enums.MessageEnum;
import com.zjl.exception.BusinessException;
import com.zjl.service.FileChunkService;
import com.zjl.service.FileService;
import com.zjl.service.LocalStorageService;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;
import sun.misc.Cleaner;

import javax.annotation.Resource;
import java.io.File;
import java.io.IOException;
import java.io.RandomAccessFile;
import java.lang.reflect.Method;
import java.nio.MappedByteBuffer;
import java.nio.channels.FileChannel;
import java.security.AccessController;
import java.security.PrivilegedAction;

/**
 * @author: zjl
 * @datetime: 2024/4/9
 * @desc:
 */
@Service("fileService")
@Slf4j
public class FileServiceImpl implements FileService {
    /**
     * 默认的分片大小：20MB
     */
    public static final long DEFAULT_CHUNK_SIZE = 20 * 1024 * 1024;

    @Value("${file.BASE_FILE_SAVE_PATH}")
    private String BASE_FILE_SAVE_PATH;

    @Resource
    private FileChunkService fileChunkService;

    @Resource
    private LocalStorageService localStorageService;

    @Override
    public boolean uploadFile(FileChunkParam param) {
        if (null == param.getFile()) {
            throw new BusinessException(MessageEnum.UPLOAD_FILE_NOT_NULL);
        }
        // 判断目录是否存在，不存在则创建目录
        File savePath = new File(BASE_FILE_SAVE_PATH);
        if (!savePath.exists()) {
            boolean flag = savePath.mkdirs();
            if (!flag) {
                log.error("保存目录创建失败");
                return false;
            }
        }


        //  todo 处理文件夹上传（上传目录下新建上传的文件夹）
        /*String relativePath = param.getRelativePath();
        if (relativePath.contains("/") || relativePath.contains(File.separator)) {
            String div = relativePath.contains(File.separator) ? File.separator : "/";
            String tempPath = relativePath.substring(0, relativePath.lastIndexOf(div));
            savePath = new File(BASE_FILE_SAVE_PATH + File.separator + tempPath);
            if (!savePath.exists()) {
                boolean flag = savePath.mkdirs();
                if (!flag) {
                    log.error("保存目录创建失败");
                    return false;
                }
            }
        }*/


        // 这里可以使用 uuid 来指定文件名，上传完成后再重命名，File.separator指文件目录分割符，win上的"\",Linux上的"/"。
        String fullFileName = savePath + File.separator + param.getFilename();
        // 单文件上传
        if (param.getTotalChunks() == 1) {
            return uploadSingleFile(fullFileName, param);
        }
        // 分片上传，这里使用 uploadFileByRandomAccessFile 方法，也可以使用 uploadFileByMappedByteBuffer 方法上传
        boolean flag = uploadFileByRandomAccessFile(fullFileName, param);
        if (!flag) {
            return false;
        }
        // 保存分片上传信息
        fileChunkService.saveFileChunk(param);
        return true;
    }


    private boolean uploadFileByRandomAccessFile(String resultFileName, FileChunkParam param) {
        try (RandomAccessFile randomAccessFile = new RandomAccessFile(resultFileName, "rw")) {
            // 分片大小必须和前端匹配，否则上传会导致文件损坏
            long chunkSize = param.getChunkSize() == 0L ? DEFAULT_CHUNK_SIZE : param.getChunkSize().longValue();
            // 偏移量
            long offset = chunkSize * (param.getChunkNumber() - 1);
            // 定位到该分片的偏移量
            randomAccessFile.seek(offset);
            // 写入
            randomAccessFile.write(param.getFile().getBytes());
        } catch (IOException e) {
            log.error("文件上传失败：" + e);
            return false;
        }
        return true;
    }

    private boolean uploadFileByMappedByteBuffer(String resultFileName, FileChunkParam param) {
        // 分片上传
        try (RandomAccessFile randomAccessFile = new RandomAccessFile(resultFileName, "rw");
             FileChannel fileChannel = randomAccessFile.getChannel()) {
            // 分片大小必须和前端匹配，否则上传会导致文件损坏
            long chunkSize = param.getChunkSize() == 0L ? DEFAULT_CHUNK_SIZE : param.getChunkSize().longValue();
            // 写入文件
            long offset = chunkSize * (param.getChunkNumber() - 1);
            byte[] fileBytes = param.getFile().getBytes();
            MappedByteBuffer mappedByteBuffer = fileChannel.map(FileChannel.MapMode.READ_WRITE, offset, fileBytes.length);
            mappedByteBuffer.put(fileBytes);
            // 释放
            unmap(mappedByteBuffer);
        } catch (IOException e) {
            log.error("文件上传失败：" + e);
            return false;
        }
        return true;
    }

    private boolean uploadSingleFile(String resultFileName, FileChunkParam param) {
        File saveFile = new File(resultFileName);
        try {
            // 写入
            param.getFile().transferTo(saveFile);
            localStorageService.saveLocalStorage(param);
        } catch (IOException e) {
            log.error("文件上传失败：" + e);
            return false;
        }
        return true;
    }

    /**
     * 释放 MappedByteBuffer
     * 在 MappedByteBuffer 释放后再对它进行读操作的话就会引发 jvm crash，在并发情况下很容易发生
     * 正在释放时另一个线程正开始读取，于是 crash 就发生了。所以为了系统稳定性释放前一般需要检
     * 查是否还有线程在读或写
     * 来源：https://my.oschina.net/feichexia/blog/212318
     *
     * @param mappedByteBuffer mappedByteBuffer
     */
    public static void unmap(final MappedByteBuffer mappedByteBuffer) {
        try {
            if (mappedByteBuffer == null) {
                return;
            }
            mappedByteBuffer.force();
            AccessController.doPrivileged((PrivilegedAction<Object>) () -> {
                try {
                    Method getCleanerMethod = mappedByteBuffer.getClass()
                            .getMethod("cleaner");
                    getCleanerMethod.setAccessible(true);
                    Cleaner cleaner =
                            (Cleaner) getCleanerMethod
                                    .invoke(mappedByteBuffer, new Object[0]);
                    cleaner.clean();
                } catch (Exception e) {
                    log.error("MappedByteBuffer 释放失败：" + e);
                }
                System.out.println("clean MappedByteBuffer completed");
                return null;
            });
        } catch (Exception e) {
            log.error("unmap error:" + e);
        }
    }
}

10.LocalStorageService

package com.zjl.service;


import com.zjl.domin.FileChunkParam;
import com.zjl.domin.LocalStorage;

import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

/**
 * @author: zjl
 * @datetime: 2024/4/9
 * @desc:
 */
public interface LocalStorageService {
    /**
     * 根据文件 md5 查询
     *
     * @param md5 md5
     * @return
     */
    LocalStorage findByMd5(String md5);

    /**
     * 保存记录
     *
     * @param localStorage 记录参数
     */
    void saveLocalStorage(LocalStorage localStorage);

    /**
     * 保存记录
     *
     * @param param 记录参数
     */
    void saveLocalStorage(FileChunkParam param);

    /**
     * 删除记录
     *
     * @param localStorage localStorage
     * @return
     */
    void delete(LocalStorage localStorage);

    /**
     * 根据 id 删除
     *
     * @param id id
     * @return
     */
    void deleteById(Long id);

    void downloadByName(String name, String md5, HttpServletRequest request, HttpServletResponse response);
}

package com.zjl.service.impl;

import com.zjl.domin.FileChunkParam;
import com.zjl.domin.LocalStorage;
import com.zjl.repository.LocalStorageRepository;
import com.zjl.service.LocalStorageService;
import com.zjl.utils.FileUtil;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;
import org.springframework.util.ObjectUtils;

import javax.annotation.Resource;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.File;
import java.io.UnsupportedEncodingException;

/**
 * @author: zjl
 * @datetime: 2024/4/9
 * @desc:
 */
@Service
@Slf4j
public class LocalStorageServiceImpl implements LocalStorageService {
    @Resource
    private LocalStorageRepository localStorageRepository;

    @Value("${file.BASE_FILE_SAVE_PATH}")
    private String BASE_FILE_SAVE_PATH;

    @Override
    public LocalStorage findByMd5(String md5) {
        return localStorageRepository.findByIdentifier(md5);
    }

    @Override
    public void saveLocalStorage(LocalStorage localStorage) {
        localStorageRepository.save(localStorage);
    }

    @Override
    public void saveLocalStorage(FileChunkParam param) {
        Long id = null;
        LocalStorage byIdentifier = localStorageRepository.findByIdentifier(param.getIdentifier());
        if (!ObjectUtils.isEmpty(byIdentifier)) {
            id = byIdentifier.getId();
        }
        String name = param.getFilename();
        String suffix = FileUtil.getExtensionName(name);
        String type = FileUtil.getFileType(suffix);
        LocalStorage localStorage = new LocalStorage(
                id,
                name,
                FileUtil.getFileNameNoEx(name),
                suffix,
                param.getRelativePath(),
                type,
                FileUtil.getSize(param.getTotalSize().longValue()),
                param.getIdentifier()
        );
        localStorageRepository.save(localStorage);
    }

    @Override
    public void delete(LocalStorage localStorage) {
        localStorageRepository.delete(localStorage);
    }

    @Override
    public void deleteById(Long id) {
        localStorageRepository.deleteById(id);
    }

    @Override
    public void downloadByName(String name, String md5, HttpServletRequest request, HttpServletResponse response) {
        LocalStorage storage = localStorageRepository.findByRealNameAndIdentifier(name, md5);
        if (ObjectUtils.isEmpty(storage)) {
            return;
        }
        File tofile = new File(BASE_FILE_SAVE_PATH + File.separator + storage.getPath());
        try {
            FileUtil.downloadFile(request, response, tofile, false);
        } catch (UnsupportedEncodingException e) {
            e.printStackTrace();
        }
    }
}

11.FileChunkService

package com.zjl.service;


import com.zjl.domin.FileChunkParam;

import java.util.List;

/**
 * @author: zjl
 * @datetime: 2024/4/9
 * @desc:
 */
public interface FileChunkService {
    /**
     * 根据文件 md5 查询
     *
     * @param md5 md5
     * @return
     */
    List<FileChunkParam> findByMd5(String md5);

    /**
     * 保存记录
     *
     * @param param 记录参数
     */
    void saveFileChunk(FileChunkParam param);

    /**
     * 删除记录
     *
     * @param fileChunk fileChunk
     * @return
     */
    void delete(FileChunkParam fileChunk);

    /**
     * 根据 id 删除
     *
     * @param id id
     * @return
     */
    void deleteById(Long id);
}

package com.zjl.service.impl;

import com.zjl.domin.FileChunkParam;
import com.zjl.repository.FileChunkRepository;
import com.zjl.service.FileChunkService;
import com.zjl.service.LocalStorageService;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

import javax.annotation.Resource;
import java.util.List;

/**
 * @author: zjl
 * @datetime: 2024/4/9
 * @desc:
 */
@Service
public class FileChunkServiceImpl implements FileChunkService {
    @Resource
    private FileChunkRepository fileChunkRepository;

    @Resource
    private LocalStorageService localStorageService;

    @Override
    public List<FileChunkParam> findByMd5(String md5) {
        return fileChunkRepository.findByIdentifier(md5);
    }

    @Override
    public void saveFileChunk(FileChunkParam param) {
        fileChunkRepository.save(param);
        // 当文件分片完整上传完成，存一份在LocalStorage表中
        if (param.getChunkNumber().equals(param.getTotalChunks())) {
            localStorageService.saveLocalStorage(param);
        }
    }

    @Override
    public void delete(FileChunkParam fileChunk) {
        fileChunkRepository.delete(fileChunk);
    }

    @Override
    public void deleteById(Long id) {
        fileChunkRepository.deleteById(id);
    }
}

12. Repository

package com.zjl.repository;

import com.zjl.domin.FileChunkParam;
import org.springframework.data.jpa.repository.JpaRepository;
import org.springframework.data.jpa.repository.JpaSpecificationExecutor;

import java.util.List;

/**
 * @author: zjl
 * @datetime: 2024/4/9
 * @desc:
 */
public interface FileChunkRepository extends JpaRepository<FileChunkParam, Long>, JpaSpecificationExecutor<FileChunkParam> {

    List<FileChunkParam> findByIdentifier(String identifier);
}

package com.zjl.repository;

import com.zjl.domin.LocalStorage;
import org.springframework.data.jpa.repository.JpaRepository;
import org.springframework.data.jpa.repository.JpaSpecificationExecutor;

/**
 * @author: zjl
 * @datetime: 2024/4/9
 * @desc:
 */
public interface LocalStorageRepository extends JpaRepository<LocalStorage, Long>, JpaSpecificationExecutor<LocalStorage> {
    LocalStorage findByIdentifier(String identifier);

    LocalStorage findByRealNameAndIdentifier(String name, String md5);
}

13.跨域配置

package com.zjl.config;

import org.springframework.context.annotation.Configuration;
import org.springframework.web.servlet.config.annotation.CorsRegistry;
import org.springframework.web.servlet.config.annotation.WebMvcConfigurer;

/**
 * @author: zjl
 * @datetime: 2024/4/9
 * @desc:
 */
@Configuration
public class GlobalCorsConfig implements WebMvcConfigurer {

    @Override
    public void addCorsMappings(CorsRegistry registry) {
        registry.addMapping("/**")
                .allowedOriginPatterns("*")
                .allowCredentials(true)
                .allowedHeaders("GET", "POST", "PUT", "DELETE", "OPTIONS")
                .allowedHeaders("Authorization", "Cache-Control", "Content-Type")
                .maxAge(3600);
    }
}

前端Vue

源码下载地址：

链接：https://pan.baidu.com/s/1KFzWdq-kfOAxMKDaCPCDPQ?pwd=6666 提取码：6666

关键代码

安装插件、指定分片大小

import SparkMD5 from "spark-md5";
const FILE_UPLOAD_ID_KEY = "file_upload_id";
// 分片大小，20MB
const CHUNK_SIZE = 20 * 1024 * 1024;

定义后端接口地址、判断分片是否上传

// 上传地址
        target: "http://127.0.0.1:9999/api/upload",
        // 是否开启服务器分片校验。默认为 true
        testChunks: true,
        // 真正上传的时候使用的 HTTP 方法,默认 POST
        uploadMethod: "post",
        // 分片大小
        chunkSize: CHUNK_SIZE,
        // 并发上传数，默认为 3
        simultaneousUploads: 3,
        /**
         * 判断分片是否上传，秒传和断点续传基于此方法
         * 这里根据实际业务来 用来判断哪些片已经上传过了 不用再重复上传了 [这里可以用来写断点续传！！！]
         */
        checkChunkUploadedByResponse: (chunk, message) => {
          // message是后台返回
          let messageObj = JSON.parse(message);
          let dataObj = messageObj.data;
          if (dataObj.uploaded !== undefined) {
            return dataObj.uploaded;
          }
          // 判断文件或分片是否已上传，已上传返回 true
          // 这里的 uploadedChunks 是后台返回]
          return (dataObj.uploadedChunks || []).indexOf(chunk.offset + 1) >= 0;
        },

计算MD5，并校验是否已上传

onFileAdded(file, event) {
      this.uploadFileList.push(file);
      console.log("file :>> ", file);
      // 有时 fileType为空，需截取字符
      console.log("文件类型：" + file.fileType);
      // 文件大小
      console.log("文件大小：" + file.size + "B");
      // 1. todo 判断文件类型是否允许上传
      // 2. 计算文件 MD5 并请求后台判断是否已上传，是则取消上传
      console.log("校验MD5");
      this.getFileMD5(file, (md5) => {
        if (md5 != "") {
          // 修改文件唯一标识
          file.uniqueIdentifier = md5;
          // 请求后台判断是否上传
          // 恢复上传
          file.resume();
        }
      });
    },

// 计算文件的MD5值
    getFileMD5(file, callback) {
      let spark = new SparkMD5.ArrayBuffer();
      let fileReader = new FileReader();
      //获取文件分片对象（注意它的兼容性，在不同浏览器的写法不同）
      let blobSlice =
        File.prototype.slice ||
        File.prototype.mozSlice ||
        File.prototype.webkitSlice;
      // 当前分片下标
      let currentChunk = 0;
      // 分片总数(向下取整)
      let chunks = Math.ceil(file.size / CHUNK_SIZE);
      // MD5加密开始时间
      let startTime = new Date().getTime();
      // 暂停上传
      file.pause();
      loadNext();
      // fileReader.readAsArrayBuffer操作会触发onload事件
      fileReader.onload = function (e) {
        // console.log("currentChunk :>> ", currentChunk);
        spark.append(e.target.result);
        if (currentChunk < chunks) {
          currentChunk++;
          loadNext();
        } else {
          // 该文件的md5值
          let md5 = spark.end();
          console.log(
            `MD5计算完毕：${md5}，耗时：${new Date().getTime() - startTime} ms.`
          );
          // 回调传值md5
          callback(md5);
        }
      };
      fileReader.onerror = function () {
        this.$message.error("文件读取错误");
        file.cancel();
      };
      // 加载下一个分片
      function loadNext() {
        const start = currentChunk * CHUNK_SIZE;
        const end =
          start + CHUNK_SIZE >= file.size ? file.size : start + CHUNK_SIZE;
        // 文件分片操作，读取下一分片(fileReader.readAsArrayBuffer操作会触发onload事件)
        fileReader.readAsArrayBuffer(blobSlice.call(file.file, start, end));
      }
    },

fileStatusText(status, response) {
      if (status === "md5") {
        return "校验MD5";
      } else {
        return this.fileStatusTextObj[status];
      }
    },

计算上传进度

onFileProgress(rootFile, file, chunk) {
      console.log(`当前进度：${Math.ceil(file._prevProgress * 100)}%`);
    },

开放原子开发者工作坊

开放原子开发者工作坊旨在鼓励更多人参与开源活动，与志同道合的开发者们相互交流开发经验、分享开发心得、获取前沿技术趋势。工作坊有多种形式的开发者活动，如meetup、训练营等，主打技术交流，干货满满，真诚地邀请各位开发者共同参与！

更多推荐

第二届开放原子大赛首批创新成果集结武汉，诚邀广大开发者共鉴开源技术盛宴

开放原子开发者工作坊

诚邀报名 | 开源基础设施能力建设分论坛：打造开源生态的“心脏”

开放原子开发者工作坊

诚邀报名 | 编程语言分论坛：AI时代的技术革新与开源实践

开放原子开发者工作坊

所有评论(0)

查看更多评论

苏生Susheng

@weixin_37833693

已为社区贡献21条内容

【SpringBoot整合系列】SpringBoot 实现大文件分片上传、断点续传及秒传

苏生Susheng

目录

功能介绍

文件上传

分片上传

秒传

断点续传

相关概念

相关方法

大文件上传流程

前端切片处理逻辑

后端处理切片的逻辑

流程解析

后端代码实现

功能目标

1.建表SQL

2.引入依赖

3.实体类

4.响应模板

5.枚举类

6.自定义异常

7.工具类

8.Controller层

9.FileService

10.LocalStorageService

11.FileChunkService

12. Repository

13.跨域配置

前端Vue

源码下载地址：

关键代码

安装插件、指定分片大小

定义后端接口地址、判断分片是否上传

计算MD5，并校验是否已上传

计算上传进度

所有评论(0)

苏生Susheng