用Hibernate插入大量数据
突然有个想法,把ip数据导入到数据库里,于是找了纯真qq的ip库,一共是331482条记录
开始的时候按常规的一条条导进去
经过观察发现save一条commit一次很慢,于是改为
速度虽然快了很多,但是依然是内存溢出
后来发现hibernate没有释放缓存,于是改成这样
这次终于可以了,331482条记录,从读取txt文件,并处理ip信息,再导入mysql的数据库,总共花了219秒
开始的时候按常规的一条条导进去
1public int save(Object transientInstance) {
2 log.debug("saving "+Object.class.toString()+" instance");
3 Transaction tx = null;
4 Session session = getSession();
5 try {
6 tx = session.beginTransaction();
7 session.save(transientInstance);
8 tx.commit();
9 log.debug("save successful");
10 return 0;
11 } catch (Exception re) {
12 log.error("save failed", re);
13 if (tx != null && tx.isActive())
14 tx.rollback();
15 return -1;
16 }
17 }
导了一个晚上还是导完,原以为再等一段时间就可以了,后来居然已内存溢出告终,真是气死。2 log.debug("saving "+Object.class.toString()+" instance");
3 Transaction tx = null;
4 Session session = getSession();
5 try {
6 tx = session.beginTransaction();
7 session.save(transientInstance);
8 tx.commit();
9 log.debug("save successful");
10 return 0;
11 } catch (Exception re) {
12 log.error("save failed", re);
13 if (tx != null && tx.isActive())
14 tx.rollback();
15 return -1;
16 }
17 }
经过观察发现save一条commit一次很慢,于是改为
1public int save(List transientInstances, int commitNum) {
2 log.debug("saving larger transientInstances");
3 Transaction tx = null;
4 Session session = getSession();
5 try {
6 tx = session.beginTransaction();
7 int i = 0;
8 for (Object transientInstance : transientInstances) {
9 session.save(transientInstance);
10 i++;
11 if (i == commitNum) {
12 tx.commit();
13 log.debug("save " + commitNum + "data successful");
14 i = 0;
15
16 tx = session.beginTransaction();
17 }
18 }
19 if (i != 0) {
20 tx.commit();
21
22 }
23 // session.
24 log.debug("save larger successful");
25 return 0;
26 } catch (Exception re) {
27 log.error("save failed", re);
28 if (tx != null && tx.isActive())
29 tx.rollback();
30 return -1;
31 }
32 }
2 log.debug("saving larger transientInstances");
3 Transaction tx = null;
4 Session session = getSession();
5 try {
6 tx = session.beginTransaction();
7 int i = 0;
8 for (Object transientInstance : transientInstances) {
9 session.save(transientInstance);
10 i++;
11 if (i == commitNum) {
12 tx.commit();
13 log.debug("save " + commitNum + "data successful");
14 i = 0;
15
16 tx = session.beginTransaction();
17 }
18 }
19 if (i != 0) {
20 tx.commit();
21
22 }
23 // session.
24 log.debug("save larger successful");
25 return 0;
26 } catch (Exception re) {
27 log.error("save failed", re);
28 if (tx != null && tx.isActive())
29 tx.rollback();
30 return -1;
31 }
32 }
速度虽然快了很多,但是依然是内存溢出
后来发现hibernate没有释放缓存,于是改成这样
1public int save(List transientInstances, int commitNum) {
2 log.debug("saving larger transientInstances");
3 Transaction tx = null;
4 Session session = getSession();
5 try {
6 tx = session.beginTransaction();
7 int i = 0;
8 for (Object transientInstance : transientInstances) {
9 session.save(transientInstance);
10 i++;
11 if (i == commitNum) {
12 tx.commit();
13 log.debug("save " + commitNum + "data successful");
14 i = 0;
15 session.flush(); session.clear();
16 tx = session.beginTransaction();
17 }
18 }
19 if (i != 0) {
20 tx.commit();
21 session.flush(); session.clear();
22 }
23 // session.
24 log.debug("save larger successful");
25 return 0;
26 } catch (Exception re) {
27 log.error("save failed", re);
28 if (tx != null && tx.isActive())
29 tx.rollback();
30 return -1;
31 }
32 }
2 log.debug("saving larger transientInstances");
3 Transaction tx = null;
4 Session session = getSession();
5 try {
6 tx = session.beginTransaction();
7 int i = 0;
8 for (Object transientInstance : transientInstances) {
9 session.save(transientInstance);
10 i++;
11 if (i == commitNum) {
12 tx.commit();
13 log.debug("save " + commitNum + "data successful");
14 i = 0;
15 session.flush(); session.clear();
16 tx = session.beginTransaction();
17 }
18 }
19 if (i != 0) {
20 tx.commit();
21 session.flush(); session.clear();
22 }
23 // session.
24 log.debug("save larger successful");
25 return 0;
26 } catch (Exception re) {
27 log.error("save failed", re);
28 if (tx != null && tx.isActive())
29 tx.rollback();
30 return -1;
31 }
32 }
这次终于可以了,331482条记录,从读取txt文件,并处理ip信息,再导入mysql的数据库,总共花了219秒