PostgreSQL9.3新增的json_populate_recordset函数使用问题

问题场景:
最近公司推出saas产品,这近半年来一直在忙于产品的转型设计工作,感觉其实搞saas比想像中的要麻烦,就目前的架构来说(不是重新设计的,只是在原来的产品架构上重构),我觉得勉强算是半个saas产品了,离真正的saas还有很大距离,总的来说比定制化开发或者说项目形式的配置化开发来得更加麻烦,也更新强调可配置性、高性能和可伸缩性。web服务器集群和后端数据库集群后,通过nginx反向代理组合成的架构的这个过程我们也遇到了很多棘手的问,也让我们学到不少东西,主要是团队里面大家基本上都是刚转型过来(从原来项目配置化应用模式到Saas应用模式),所以难免磕磕碰碰。而今天我主要思考的是数据库集群过程中遇到的问题,首先我们的数据库集群大致如下:
说明:
1.postgresql数据库复制结构分为主从库,如上图中,master为主库,主库可以处理读写的事务,slave为从库,从库只处理只读事务;
2.复制架构可以配置为一台主机为主库,多台其他主机为从库(估计是从库读取数据实际上比写入更加耗费资源);
3.主从库之间的视图是不同时刻的视图,也就是说视图无法在主从或者多台从库之前实现同步;
4.主从库的数据同步是通过日志运行来实现,并且是单向的,只能从主库通过日志分发到多台从库中;
问题描述:
1.如上图(详情可参考PostgreSQL-XC),我们saas产品一个核心功能就是配置化,在配置化功能中,由于目前面向的主要快些行业,所以系统中的一个核心功能是报表配置化,但是还是所以业务系统的报表,而业务系统的特征就是日志的业务基础数据量是异常的大,比如哇哈哈的报表,每天就要求从一个报表导出几万数据(原则上已经背离报表的使用场景),所以数据度要求读写分离要实现高性能,但是使用PostgreSQL只能用函数当作存贮过程,所以每个函数写法基本都是这样子:
select * from 函数名(...,游标);
fetch all from 游标;
这样的语句在程序里面无法判断是读还是写,(因为在函数里面报表经常要创建临时表,这在只读库的事物里面是不运行的)否则就报错如下:
ERROR:  cannot execute INSERT in a read-only transaction
先问题来了,项目里面最吃性能的现在主要就是报表这一块了,请求数据量太大了,所以才设计了读写分离的库,可是如果没法实现可以在报表函数里面创建临时表,那读写分离就毫无意义了(由于测试发现了这个BUG,所以目前全部都暂时用主数据库去操作)。那我应该如何解决这个问题呢?所以我提出下面的一个假设来解决问题这个,以及我对这个假设的解决方案的一个论证过程,当做我学习的一点一滴的积累吧。
 
方案假设:
1.使用postgres9.2新增的json数据类型来解决在只读事物中不能创建临时表的问题。
 
方案论证:
1.经过研究发现,虽然pg9.2中提供json类型,也可以讲数据行转化为数组json存放,但是却不能从数组json中反过来转化为数据行,所以使用9.2json类型来解决这个问题的思路一下打断了,想了想,目前postgres9.2已经是最新的发布版本了,最新版本都没有方案解决,我是不是就无可奈何了呢,可是我还是不服气,于是查阅了官方网站的最新用户手册,惊奇发现PostgreSQL 9.3 Beta 1 Released! (2013-05-13发布的),然后看了它的手册,发现有个函数可以将数组json中反过来转化为数据行,一下高兴了,可选没有安装包,只有源码!不过还好我在最近不用windows了(虽然在.NET下开发,但是我越来越讨厌微软了,啥都收钱,连我的最爱的VS都不给用,这由于公司即将上市的原因,微软来“打劫”了,我的vs也改用了开源的SharpDevelop 4.3),改用ubuntu13.4,所以下载了源码从新编译安装,过来一个小时,终于搞定了,下面是我的论证过程:
参考资料:
首先来试试json功能,体验体验
--原始查询
 select a.xwname,a.xwsex,a.xwage,b.xwadress from test_t_a a
 inner join test_t_b b on a.xwfid=b.xwfid
 
--原始行全部输出
 select row_to_json(v) from (
 select a.xwname,a.xwsex,a.xwage,b.xwadress from test_t_a a
 inner join test_t_b b on a.xwfid=b.xwfid
 ) v
 
--筛选行,使用动态列名
 select row_to_json(row(xwname, xwadress)) from (
 select a.xwname,a.xwsex,a.xwage,b.xwadress from test_t_a a
 inner join test_t_b b on a.xwfid=b.xwfid
 ) v2
--筛选行,使用原始列名,输出成数组
select array_to_json(array_agg(row_to_json(t)))
   from (
        select a.xwname,a.xwsex,a.xwage,b.xwadress from test_t_a a
        inner join test_t_b b on a.xwfid=b.xwfid
   ) t
--把数组json转换成数据行
 
select * from json_populate_recordset(row(a::text, b::text), '[{"a":1,"b":2},{"a":3,"b":4}]')
现在好戏开场了:(由于我的台式机ubuntu里面没安装"天朝"输入法,只能将就打“拼音”了,测试过程的截图,我是放在wiznote笔记里面记录,然后空闲的时候用笔记本整理好发布到博客上来
================9.3test the new function of json=====================================================

1.  the following picture show the 9.3 version of postgresql database provide a number of   json functions:

now, i try it on what thoes json function can do.firt,we need to create two table for testing,which show with the following:

-- Table: table_users

 

-- DROP TABLE table_users;

 

CREATE TABLE table_users

(

 xwid uuid NOT NULL,

 xwname character varying,

 xwage integer,

 xwfid uuid,

 CONSTRAINT table_user_pk PRIMARY KEY (xwid),

 CONSTRAINT table_users_fk FOREIGN KEY (xwfid)

     REFERENCES table_adress (xwfid) MATCH Unknown

     ON UPDATE NO ACTION ON DELETE NO ACTION

)

WITH (

 OIDS=FALSE

);

ALTER TABLE table_users

 OWNER TO postgres;

 

-- Index: fki_table_users_fk

 

-- DROP INDEX fki_table_users_fk;

 

CREATE INDEX fki_table_users_fk

 ON table_users

 USING btree

 (xwfid);

================================================================

-- Table: table_adress

 

-- DROP TABLE table_adress;

 

CREATE TABLE table_adress

(

 xwfid uuid NOT NULL,

 xwadress character varying,

 CONSTRAINT table_adress_pk PRIMARY KEY (xwfid)

)

WITH (

 OIDS=FALSE

);

ALTER TABLE table_adress

 OWNER TO postgres;

================================================================

now we need to insert some test data into the two tables;

--insert into the table_adress table

INSERT INTO table_adress(xwfid, xwadress)VALUES ('5b9898f7-4456-4390-be2b-525c3274855a','China');

INSERT INTO table_adress(xwfid, xwadress)VALUES ('6b9898f7-4456-4390-be2b-525c3274855a','Japan');

INSERT INTO table_adress(xwfid, xwadress)VALUES ('7b9898f7-4456-4390-be2b-525c3274855a','America');

INSERT INTO table_adress(xwfid, xwadress)VALUES ('8b9898f7-4456-4390-be2b-525c3274855a','Russia');

 

--insert into the table_users table

INSERT INTO table_users(xwid, xwname, xwage, xwfid)

VALUES ('1a9898f7-4456-4390-be2b-525c3274855a','成龙',50,'5b9898f7-4456-4390-be2b-525c3274855a');

INSERT INTO table_users(xwid, xwname, xwage, xwfid)

VALUES ('2a9898f7-4456-4390-be2b-525c3274855a','蒼井空',28,'6b9898f7-4456-4390-be2b-525c3274855a');

INSERT INTO table_users(xwid, xwname, xwage, xwfid)

VALUES ('3a9898f7-4456-4390-be2b-525c3274855a','Nicolas Cage',49,'7b9898f7-4456-4390-be2b-525c3274855a');

INSERT INTO table_users(xwid, xwname, xwage, xwfid)

VALUES ('4a9898f7-4456-4390-be2b-525c3274855a','Влади́мир Влади́мирович Пу́тин', 61,'8b9898f7-4456-4390-be2b-525c3274855a');

================================================================

select * from table_adress;

resulte:

select * from table_users;

resulte:

select array_to_json(array_agg(row_to_json(t)))

   from (

        select u.xwname,u.xwage,a.xwadress from table_users as u,table_adress as a where u.xwfid=a.xwfid

   ) t

--resulte:'[{"xwname":"成龙","xwage":50,"xwadress":"China"},

--          {"xwname":"蒼井空","xwage":28,"xwadress":"Japan"},

--          {"xwname":"Nicolas Cage","xwage":49,"xwadress":"America"},

--          {"xwname":"Влади́мир Влади́мирович Пу́тин","xwage":61,"xwadress":"Russia"}]'

==============================================================================

select * from     json_populate_recordset(null::t_row,'[{"a":1,"b":2},{"a":3,"b":4}]')

ERROR:  type "t_row" does not exist

LINE 1: select * from     json_populate_recordset(null::t_row,'[{"a":...

********** Error **********

 

ERROR: type "t_row" does not exist

SQL state: 42704

Character: 48

=============================================================================

CREATE OR REPLACE FUNCTION get_jsonrows(a refcursor)

 RETURNS SETOF refcursor AS

$BODY$          

DECLARE

msgjson character varying :='';

message_row record;

BEGIN  

       select u.xwname,u.xwage,a.xwadress from table_users as u,table_adress as a where false limit 1 into message_row;

       select array_to_json(array_agg(row_to_json(t)))

       from (

       select u.xwname,u.xwage,a.xwadress from table_users as u,table_adress as a where u.xwfid=a.xwfid

       ) t into msgjson;

 

--OPEN a FOR select * from json_populate_recordset(null::message_row,msgjson);

OPEN a FOR select msgjson as msgjson;

RETURN NEXT a;

END;

$BODY$

 LANGUAGE plpgsql VOLATILE

 COST 100

====test:resulte

select * from get_jsonrows('a');

fetch all from a;

===data:

'[{"xwname":"成龙","xwage":50,"xwadress":"China"},{"xwname":"蒼井空","xwage":28,"xwadress":"Japan"},{"xwname":"Nicolas Cage","xwage":49,"xwadress":"America"},{"xwname":"Влади́мир Влади́мирович Пу́тин","xwage":61,"xwadress":"Russia"}]'

========================================================================================================

CREATE OR REPLACE FUNCTION get_jsonrows(a refcursor)

 RETURNS SETOF refcursor AS

$BODY$          

DECLARE

msgjson character varying :='';

message_row record;

BEGIN  

       select u.xwname,u.xwage,a.xwadress from table_users as u,table_adress as a where false limit 1 into message_row;

       select array_to_json(array_agg(row_to_json(t)))

       from (

       select u.xwname,u.xwage,a.xwadress from table_users as u,table_adress as a where u.xwfid=a.xwfid

       ) t into msgjson;

 

OPEN a FOR select * from json_populate_recordset(null::message_row,msgjson);

--OPEN a FOR select msgjson as msgjson;

RETURN NEXT a;

END;

$BODY$

 LANGUAGE plpgsql VOLATILE

 COST 100

===========test:resulte

select * from get_jsonrows('a');

fetch all from a;

====data:

ERROR:  type "message_row" does not exist

LINE 1: select * from json_populate_recordset(null::message_row,msgj...

                                                   ^

QUERY:  select * from json_populate_recordset(null::message_row,msgjson)

CONTEXT:  PL/pgSQL function get_jsonrows(refcursor) line 12 at OPEN

 

********** Error **********

 

ERROR: type "message_row" does not exist

SQL state: 42704

Context: PL/pgSQL function get_jsonrows(refcursor) line 12 at OPEN

======================================================================

CREATE OR REPLACE FUNCTION get_jsonrows(a refcursor)

 RETURNS SETOF refcursor AS

$BODY$          

DECLARE

msgjson character varying :='';

message_row record;

BEGIN  

       select u.xwname,u.xwage,a.xwadress from table_users as u,table_adress as a where false limit 1 into message_row;

       select array_to_json(array_agg(row_to_json(t)))

       from (

       select u.xwname,u.xwage,a.xwadress from table_users as u,table_adress as a where u.xwfid=a.xwfid

       ) t into msgjson;

 

--OPEN a FOR select * from json_populate_recordset(null::message_row,msgjson);

--OPEN a FOR select msgjson as msgjson;

OPEN a FOR select message_row.xwname as xwname,message_row.xwage as xwage,message_row.xwadress as xwadress;

RETURN NEXT a;

END;

$BODY$

 LANGUAGE plpgsql VOLATILE

 COST 100

论证结果:
1.悲剧+悲伤,论证结果为:no pass!!!!
结论是: json_populate_recordset函数的第一个参数的rowtype必须为物理表的结构.
不过没关系,重在过程和发问,多走走弯路就多学到东西。
 
 
 
 





posted @ 2013-06-18 22:58  正在获取...  阅读(3184)  评论(0编辑  收藏  举报