UTF-8フラグを落とすのはどれが早いのか 
Wednesday, January 26, 2011, 07:49 AM - Programing, Programing / Perl
Posted by Administrator
Perl 5.8以降導入されたUTF-8フラグを落とすのはどれが早いのか

#!/usr/bin/perl
use Benchmark;
use Encode;

while(<STDIN>){
$gplv2 .= $_;
}
utf8::decode($gplv2);

timethese(
1000000,
{
"utf8::encode" =>sub(){
my $s = $gplv2;
utf8::encode($s);
return();
},
"Encode::encode" =>sub(){
my $s = Encode::encode("utf8",$gplv2);
return();
},
"Encode::encode_utf8" =>sub(){
my $s = Encode::encode_utf8($gplv2);
return();
}
}
);

STDINへぶち込むのはGPLv2 日本語訳全文
> Benchmark: timing 1000000 iterations of Encode::encode, Encode::encode_utf8, utf8::encode...
> Encode::encode: 14 wallclock secs (11.13 usr + 0.01 sys = 11.14 CPU) @ 89766.61/s (n=1000000)
> Encode::encode_utf8: 7 wallclock secs ( 5.56 usr + 0.01 sys = 5.57 CPU) @ 179533.21/s (n=1000000)
> utf8::encode: 4 wallclock secs ( 2.41 usr + 0.00 sys = 2.41 CPU) @ 414937.76/s (n=1000000)
圧勝過ぎワロタ
1 comment ( 1411 views )   |  permalink
DBI->connect_cached(...)は早いの? 
Tuesday, January 18, 2011, 06:12 PM - Programing, Programing / Perl
Posted by Administrator
DBIのpodにも書いてあるけど、->*_cachedは遅い
多少内部処理がお粗末になる覚悟でそれっぽいモジュールや後述するplan Aのような単純な実装を選択する方が望ましいだろう

検証用コード
#!/usr/bin/perl
use Benchmark;
use DBI;

sub plan_a
{
$sth1->bind_param(1,$ARGV[1],DBI::SQL_INTEGER);
$sth1->execute();
while($sth1->fetch()){
}
$sth1->finish();

return();
}

sub plan_b
{
my $dbh2 = DBI->connect(qw(dbi:mysql:database=a;host=172.16.2.205;port=3306 a));
my $sth2 = $dbh2->prepare("SELECT * FROM `a` WHERE 1 LIMIT 0,?");
$sth2->bind_param(1,$ARGV[1],DBI::SQL_INTEGER);
$sth2->execute();
while($sth2->fetch()){
}
$sth2->finish();
$dbh2->disconnect();

return();
}

sub plan_c
{
my $dbh3 = DBI->connect_cached(qw(dbi:mysql:database=a;host=172.16.2.205;port=3306 a));
my $sth3 = $dbh3->prepare_cached("SELECT * FROM `a` WHERE 1 LIMIT 0,?");
$sth3->bind_param(1,$ARGV[1],DBI::SQL_INTEGER);
$sth3->execute();
while($sth3->fetch()){
}

return();
}

printf("=== %d records, %d repeats ===\n",reverse(@ARGV));

$dbh1 = DBI->connect(qw(dbi:mysql:database=a;host=172.16.2.205;port=3306 a));
$sth1 = $dbh1->prepare("SELECT * FROM `a` WHERE 1 LIMIT 0,?");

sqrt($_) for(1..10000);
timethese(
$ARGV[0],
{
"A" =>\&plan_a,
"B" =>\&plan_b,
"C" =>\&plan_c,
}
);

__END__

の結果
> === 1 records, 10000 repeats ===
> Benchmark: timing 10000 iterations of A, B, C...
> A: 6 wallclock secs ( 1.23 usr + 0.34 sys = 1.57 CPU) @ 6369.43/s (n=10000)
> B: 41 wallclock secs ( 8.22 usr + 2.34 sys = 10.56 CPU) @ 946.97/s (n=10000)
> C: 17 wallclock secs ( 4.84 usr + 0.74 sys = 5.58 CPU) @ 1792.11/s (n=10000)
これは少々酷い
接続切断を繰り返すplan Bと比べれば2倍だろうが、plan Aから見れば大きな変化とは言えない

大体こういうふざけたモジュールが存在していたとして
package CachedDBI4;
use DBI;
use Data::Dumper;
use vars qw(%db_vars);
use vars qw(%st_vars);

sub connect_cached
{
my($f,@r) = @_;

return $CachedDBI4::db_vars{Dumper(@r)} ||= $f->connect(@r);
}


sub prepare_cached
{
my($f,@r) = @_;

return $CachedDBI4::st_vars{Dumper(@r)} ||= $f->prepare(@r);
}

*DBI::connect_cached = \&connect_cached;
*DBI::db::prepare_cached = \&prepare_cached;

__PACKAGE__

これを使ったスコアが
> === 1 records, 10000 repeats ===
> Benchmark: timing 10000 iterations of D...
> D: 9 wallclock secs ( 3.94 usr + 0.49 sys = 4.43 CPU) @ 2257.34/s (n=10000)
B, C, Dとどんぐりだが、こんなコードが最も早いとは思わなかったわけで

podに書いている時点で知れてる事だけど
add comment ( 710 views )   |  permalink
BlackCurtain::DBI::Cache::Memcachedって早いの? (2) 
Thursday, January 13, 2011, 12:10 PM - Programing, Programing / Perl
Posted by Administrator
実のところBlackCurtain::DBI::Cache::Memcachedが遅い、加えてその検証結果も予定調和だったりして

検証用コードは前回検証の付録と同一(timetheseの直前でCool'n'Quiteの影響を緩和するため糞ループを追加している)
sub BlackCurtainDBI
{
$sth2->bind_param(1,$ARGV[1],DBI::SQL_INTEGER);
$sth2->execute();
while($sth2->fetch()){
}
$sth2->finish();

return();
}

結果
> === 1 records, 100000 repeats ===
> Benchmark: timing 100000 iterations of BCDBI(a)...
> BCDBI(a): 14 wallclock secs ( 7.73 usr + 2.12 sys = 9.85 CPU) @ 10152.28/s (n=100000)
> === 10 records, 100000 repeats ===
> Benchmark: timing 100000 iterations of BCDBI(a)...
> BCDBI(a): 19 wallclock secs (13.48 usr + 1.76 sys = 15.24 CPU) @ 6561.68/s (n=100000)
> === 100 records, 100000 repeats ===
> Benchmark: timing 100000 iterations of BCDBI(a)...
> BCDBI(a): 71 wallclock secs (62.88 usr + 3.10 sys = 65.98 CPU) @ 1515.61/s (n=100000)
この傾向は前回の検証でも十分観測できる

と、いうことはBlackCurtain::DBI::Cache::Memcached::stのsub FETCHが高い確率で悪
邪道だが動作そのものに問題ないので削除

そして結果
> === 1 records, 100000 repeats ===
> Benchmark: timing 100000 iterations of BCDBI(a)...
> BCDBI(a): 9 wallclock secs ( 7.51 usr + 0.87 sys = 8.38 CPU) @ 11933.17/s (n=100000)
> === 10 records, 100000 repeats ===
> Benchmark: timing 100000 iterations of BCDBI(a)...
> BCDBI(a): 16 wallclock secs (10.18 usr + 1.83 sys = 12.01 CPU) @ 8326.39/s (n=100000)
> === 100 records, 100000 repeats ===
> Benchmark: timing 100000 iterations of BCDBI(a)...
> BCDBI(a): 58 wallclock secs (49.93 usr + 1.05 sys = 50.98 CPU) @ 1961.55/s (n=100000)
正直コード削れば早くなるのは実に当たり前の話だがな
ともかく今回に限らずsub FETCH{ ... }へ割り込むのは余り好ましくないっていうのははっきり見える結果

そしてこれを踏襲し
・ FETCH/STOREは最小限にしろ
・ サブクラスから->SUPERするな
・ コードを書くな
の3点を重点に置いて考えるならば

->execute(...)で勝手に全部キャッシュに突っ込んで->fetch(...)で->SUPER::fetch(...)しない
こんな感じのコードが好ましい
sub execute
{
my($f,@r) = @_;

my $memcached = \$f->{Database}->{Memcached_};
my $cache = \$f->{Memcached_cache};
(my $q = $f->{Statement}) =~s/\?/$f->{ParamValues}->{my $i++}/go;
if(!($$cache = $$memcached->get($q))){
$f->SUPER::execute();
push(@{$$cache},$row) while(my $row = $f->SUPER::fetch());
$$memcached->set($q,\@$$cache);

}
return($#{$$cache});
}

sub fetch
{
my($f,@r) = @_;

return(shift(@{$f->{Memcached_cache}}));
}

結果(5回試行して最高と最低を記載、どうしてか揺れる)
> === 1 records, 100000 repeats ===
> Benchmark: timing 100000 iterations of BCDBI(a)...
> BCDBI(a): 7 wallclock secs ( 4.65 usr + 0.84 sys = 5.49 CPU) @ 18214.94/s (n=100000)
> === 1 records, 100000 repeats ===
> Benchmark: timing 100000 iterations of BCDBI(a)...
> BCDBI(a): 11 wallclock secs ( 5.28 usr + 2.16 sys = 7.44 CPU) @ 13440.86/s (n=100000)
> === 10 records, 100000 repeats ===
> Benchmark: timing 100000 iterations of BCDBI(a)...
> BCDBI(a): 7 wallclock secs ( 4.74 usr + 0.86 sys = 5.60 CPU) @ 17857.14/s (n=100000)
> === 10 records, 100000 repeats ===
> Benchmark: timing 100000 iterations of BCDBI(a)...
> BCDBI(a): 12 wallclock secs ( 5.41 usr + 2.25 sys = 7.66 CPU) @ 13054.83/s (n=100000)
> === 100 records, 100000 repeats ===
> Benchmark: timing 100000 iterations of BCDBI(a)...
> BCDBI(a): 8 wallclock secs ( 5.41 usr + 0.89 sys = 6.30 CPU) @ 15873.02/s (n=100000)
> === 100 records, 100000 repeats ===
> Benchmark: timing 100000 iterations of BCDBI(a)...
> BCDBI(a): 10 wallclock secs ( 5.54 usr + 1.67 sys = 7.21 CPU) @ 13869.63/s (n=100000)
1.0bと比べると理想的なカーブでパフォーマンスが向上している

pDBIが1, 10, 100 recordsで7739.94/s, 4095.00/s, 880.98/sである事を考えると13000/s以上の速度は上々といったところか
add comment ( 1556 views )   |  permalink
BlackCurtain::DBI::Cache::Memcachedって早いの? 
Tuesday, January 11, 2011, 04:43 AM - Programing, Programing / Perl
Posted by Administrator
BlackCurtain::DBI::Cache::Memcachedは自分用途(も当然あるが)ではなく業務用途なので速度を気にする必要がある
1.0bはMemcachedこそ使うが、恩恵による速度向上と拡張による速度低下のバランスが麗しくないので、その検証

実験用のコードはこれ
#!/usr/bin/perl
use Benchmark;
use DBI;
use BlackCurtain::DBI::Cache::Memcached;

sub perlDBI
{
$dbh = DBI->connect(qw(dbi:mysql:database=a;host=172.16.2.205;port=3306 a));
$sth = $dbh->prepare("SELECT * FROM `a` WHERE 1 LIMIT 0,?");
$sth->bind_param(1,$ARGV[1],DBI::SQL_INTEGER);
$sth->execute();
while($sth->fetch()){
}
$sth->finish();
$dbh->disconnect();

return();
}

sub BlackCurtainDBI
{
$dbh = BlackCurtain::DBI::Cache::Memcached->connect(qw(dbi:mysql:database=a;host=172.16.2.205;port=3306 a),undef,{Memcached =>{servers =>[qw(127.0.0.1:11211)]}});
$sth = $dbh->prepare("SELECT * FROM `a` WHERE 1 LIMIT 0,?");
$sth->bind_param(1,$ARGV[1],DBI::SQL_INTEGER);
$sth->execute();
while($sth->fetch()){
}
$sth->finish();
$dbh->disconnect();

return();
}

sub BlackCurtainDBI_noncache
{
$dbh = BlackCurtain::DBI::Cache::Memcached->connect(qw(dbi:mysql:database=a;host=172.16.2.205;port=3306 a));
$sth = $dbh->prepare("SELECT * FROM `a` WHERE 1 LIMIT 0,?");
$sth->bind_param(1,$ARGV[1],DBI::SQL_INTEGER);
$sth->execute();
while($sth->fetch()){
}
$sth->finish();
$dbh->disconnect();

return();
}

printf("=== %d records, %d repeats ===\n",reverse(@ARGV));

timethese(
$ARGV[0],
{
"pDBI" =>\&perlDBI,
"BCDBI(a)" =>\&BlackCurtainDBI,
"BCDBI(b)" =>\&BlackCurtainDBI_noncache,
}
);

__END__

で、測定
(a)はMemcachedを使用、(b)は使用していない
> === 10 records, 100 repeats ===
> Benchmark: timing 100 iterations of BCDBI(a), BCDBI(b), pDBI...
> BCDBI(a): 0 wallclock secs ( 0.17 usr + 0.04 sys = 0.21 CPU) @ 476.19/s (n=100)
> BCDBI(b): 1 wallclock secs ( 0.15 usr + 0.03 sys = 0.18 CPU) @ 555.56/s (n=100)
> pDBI: 0 wallclock secs ( 0.10 usr + 0.02 sys = 0.12 CPU) @ 833.33/s (n=100)
> === 100 records, 100 repeats ===
> Benchmark: timing 100 iterations of BCDBI(a), BCDBI(b), pDBI...
> BCDBI(a): 1 wallclock secs ( 0.18 usr + 0.03 sys = 0.21 CPU) @ 476.19/s (n=100)
> BCDBI(b): 1 wallclock secs ( 0.65 usr + 0.04 sys = 0.69 CPU) @ 144.93/s (n=100)
> pDBI: 1 wallclock secs ( 0.18 usr + 0.03 sys = 0.21 CPU) @ 476.19/s (n=100)
> === 1000 records, 100 repeats ===
> Benchmark: timing 100 iterations of BCDBI(a), BCDBI(b), pDBI...
> BCDBI(a): 1 wallclock secs ( 0.69 usr + 0.02 sys = 0.71 CPU) @ 140.85/s (n=100)
> BCDBI(b): 3 wallclock secs ( 2.02 usr + 0.06 sys = 2.08 CPU) @ 48.08/s (n=100)
> pDBI: 3 wallclock secs ( 0.97 usr + 0.13 sys = 1.10 CPU) @ 90.91/s (n=100)
> === 10000 records, 100 repeats ===
> Benchmark: timing 100 iterations of BCDBI(a), BCDBI(b), pDBI...
> BCDBI(a): 6 wallclock secs ( 5.93 usr + 0.04 sys = 5.97 CPU) @ 16.75/s (n=100)
> BCDBI(b): 27 wallclock secs (17.03 usr + 0.87 sys = 17.90 CPU) @ 5.59/s (n=100)
> pDBI: 15 wallclock secs ( 4.98 usr + 0.83 sys = 5.81 CPU) @ 17.21/s (n=100)
> === 100000 records, 100 repeats ===
> Benchmark: timing 100 iterations of BCDBI(a), BCDBI(b), pDBI...
> BCDBI(a): 58 wallclock secs (56.26 usr + 0.22 sys = 56.48 CPU) @ 1.77/s (n=100)
> BCDBI(b): 240 wallclock secs (144.47 usr + 13.74 sys = 158.21 CPU) @ 0.63/s (n=100)
> pDBI: 131 wallclock secs (34.73 usr + 13.61 sys = 48.34 CPU) @ 2.07/s (n=100)
Memcachedを組み込む事で評判通り爆発的な速度が見込める事がわかる
拡張による速度低下も凄まじい(2~3倍)が、それを相殺しきれているのだから驚く
しかし(非現実的だが)10000レコードまで膨らむと1.0aではどうしようもなくなる、こんなの当たり前の有り得ない話(sub fetch{ ... }を見たまえ、こんなコードで10000レコードを扱えるわけがない)だが

ともかくMemcachedはぶっ飛ぶ程早い
では現状だとキャッシュ利用による速度向上幅はどの程度なのか
> === 10 records, 1000 repeats ===
> Benchmark: timing 1000 iterations of BCDBI(a), BCDBI(b), pDBI...
> BCDBI(a): 5 wallclock secs ( 1.82 usr + 0.53 sys = 2.35 CPU) @ 425.53/s (n=1000)
> BCDBI(b): 6 wallclock secs ( 2.03 usr + 0.30 sys = 2.33 CPU) @ 429.18/s (n=1000)
> pDBI: 5 wallclock secs ( 1.12 usr + 0.28 sys = 1.40 CPU) @ 714.29/s (n=1000)
> === 100 records, 1000 repeats ===
> Benchmark: timing 1000 iterations of BCDBI(a), BCDBI(b), pDBI...
> BCDBI(a): 5 wallclock secs ( 2.08 usr + 0.21 sys = 2.29 CPU) @ 436.68/s (n=1000)
> BCDBI(b): 9 wallclock secs ( 4.37 usr + 0.23 sys = 4.60 CPU) @ 217.39/s (n=1000)
> pDBI: 7 wallclock secs ( 1.97 usr + 0.35 sys = 2.32 CPU) @ 431.03/s (n=1000)
> === 1000 records, 1000 repeats ===
> Benchmark: timing 1000 iterations of BCDBI(a), BCDBI(b), pDBI...
> BCDBI(a): 10 wallclock secs ( 6.52 usr + 0.16 sys = 6.68 CPU) @ 149.70/s (n=1000)
> BCDBI(b): 36 wallclock secs (21.83 usr + 0.57 sys = 22.40 CPU) @ 44.64/s (n=1000)
> pDBI: 25 wallclock secs ( 9.88 usr + 1.32 sys = 11.20 CPU) @ 89.29/s (n=1000)
> === 10 records, 10000 repeats ===
> Benchmark: timing 10000 iterations of BCDBI(a), BCDBI(b), pDBI...
> BCDBI(a): 50 wallclock secs (19.32 usr + 3.02 sys = 22.34 CPU) @ 447.63/s (n=10000)
> BCDBI(b): 58 wallclock secs (19.18 usr + 2.28 sys = 21.46 CPU) @ 465.98/s (n=10000)
> pDBI: 46 wallclock secs ( 9.55 usr + 2.36 sys = 11.91 CPU) @ 839.63/s (n=10000)
> === 100 records, 10000 repeats ===
> Benchmark: timing 10000 iterations of BCDBI(a), BCDBI(b), pDBI...
> BCDBI(a): 47 wallclock secs (19.68 usr + 1.75 sys = 21.43 CPU) @ 466.64/s (n=10000)
> BCDBI(b): 110 wallclock secs (59.99 usr + 2.84 sys = 62.83 CPU) @ 159.16/s (n=10000)
> pDBI: 68 wallclock secs (18.11 usr + 3.25 sys = 21.36 CPU) @ 468.16/s (n=10000)
と、いうような結果になった
これは"恩恵による速度向上=拡張による速度低下"ということ
1.0bはまるで意味がない

付録(運用予定のセッション維持なコードを使ったベンチマーク)はREADMORE

Read More...
add comment ( 634 views )   |  permalink
BlackCurtain-DBI-Cache-Memcached-1.0b.tar.gz 
Monday, January 10, 2011, 09:41 AM - Release
Posted by Administrator
Archive : BlackCurtain-DBI-Cache-Memcached-1.0b.tar.gz (755B)

DBIそのままでMemcachedの恩恵を受けられるモジュール……の試作版

従来のDBIなコード
my $dbh = DBI->connect(qw(dbi:mysql:database=database;host=127.0.0.1;port=3306 user pass));

my $dbh = BlackCurtain::DBI::Cache::Memcached->connect(qw(dbi:mysql:database=database;host=127.0.0.1;port=3306 user pass),{Memcached =>{servers =>[qw(127.0.0.1:11211)]}});
と、変更するだけ

※注意
試作版なので恩恵によるパフォーマンス向上がプラスであるとは限らない件について
add comment ( 1607 views )   |  permalink

<<First <Back | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | Next> Last>>