Проблема/непонимание работы сборщика мусора или утечка памяти

@main[]
^for[i](0;100){
	^import[]
}

@import[][messages]
^for[j](0;16){
	$messages[^ImportClassObject.import_by_part[$j]]
	$messages[]
	^Rusage:compact[]
}
$result[]

[BEFORE 1] {"used" => "4524", "free" => "252", "ever_allocated_since_compact" => "2225.3", "ever_allocated_since_start" => "8130.47"}
	[BEFORE load] {"used" => "4536", "free" => "240", "ever_allocated_since_compact" => "2260.33", "ever_allocated_since_start" => "8165.5"}
	[AFTER load] {"used" => "14648", "free" => "3620", "ever_allocated_since_compact" => "57827.6", "ever_allocated_since_start" => "306299"}
	[AFTER import] {"used" => "24584", "free" => "7900", "ever_allocated_since_compact" => "14020.2", "ever_allocated_since_start" => "658280"}
[BEFORE 2] {"used" => "24608", "free" => "7876", "ever_allocated_since_compact" => "14933.9", "ever_allocated_since_start" => "659194"}
	[BEFORE load] {"used" => "24608", "free" => "7876", "ever_allocated_since_compact" => "14967.8", "ever_allocated_since_start" => "659228"}
	[AFTER load] {"used" => "31188", "free" => "1296", "ever_allocated_since_compact" => "28604.2", "ever_allocated_since_start" => "957702"}
	[AFTER import] {"used" => "34736", "free" => "5940", "ever_allocated_since_compact" => "10538.1", "ever_allocated_since_start" => "1.31113e+06"}
[BEFORE 3] {"used" => "34760", "free" => "5916", "ever_allocated_since_compact" => "11452.9", "ever_allocated_since_start" => "1.31204e+06"}
	[BEFORE load] {"used" => "34760", "free" => "5916", "ever_allocated_since_compact" => "11486.9", "ever_allocated_since_start" => "1.31207e+06"}
	[AFTER load] {"used" => "41064", "free" => "7804", "ever_allocated_since_compact" => "154914", "ever_allocated_since_start" => "1.60606e+06"}
	[AFTER import] {"used" => "44200", "free" => "4668", "ever_allocated_since_compact" => "28073", "ever_allocated_since_start" => "1.96236e+06"}

Что-то много памяти расходуется ...

[BEFORE load] {"used" => "4508", "free" => "268", "ever_allocated_since_compact" => "2200.77", "ever_allocated_since_start" => "8103.62"}
		[BEFORE _request API] {"used" => "4516", "free" => "260", "ever_allocated_since_compact" => "2233.22", "ever_allocated_since_start" => "8136.08"}
		[AFTER _request API] {"used" => "11200", "free" => "2508", "ever_allocated_since_compact" => "9954.66", "ever_allocated_since_start" => "15857.5"}
		[AFTER xml2hash] {"used" => "14024", "free" => "4264", "ever_allocated_since_compact" => "57555.8", "ever_allocated_since_start" => "304741"}
[AFTER load] {"used" => "14512", "free" => "3776", "ever_allocated_since_compact" => "59245.5", "ever_allocated_since_start" => "306431"}
	[AFTER import] {"used" => "25168", "free" => "7352", "ever_allocated_since_compact" => "20328.9", "ever_allocated_since_start" => "669705"}
[BEFORE load] {"used" => "25392", "free" => "7128", "ever_allocated_since_compact" => "21218.2", "ever_allocated_since_start" => "670594"}
		[BEFORE _request API] {"used" => "25396", "free" => "7124", "ever_allocated_since_compact" => "21250.5", "ever_allocated_since_start" => "670626"}
		[AFTER _request API] {"used" => "32552", "free" => "8160", "ever_allocated_since_compact" => "29154", "ever_allocated_since_start" => "678530"}
		[AFTER xml2hash] {"used" => "31844", "free" => "8868", "ever_allocated_since_compact" => "16770.5", "ever_allocated_since_start" => "967607"}
[AFTER load] {"used" => "31928", "free" => "8784", "ever_allocated_since_compact" => "18459.7", "ever_allocated_since_start" => "969296"}

##############################################################################
@_xml2hash[xNode]
	^switch($xNode.nodeType){
		^case(1){
			$result[^hash::create[]]
			
			$list[^xNode.select[*]]
			
			^switch(true){
				^case($xNode.nodeName eq "ids"){
					^for[i](0;$list-1){
						$node[$list.$i]

						$result.[$i][^_xml2hash[$node]]
					}
				}
				
				^case(^xNode.getAttribute[type] eq "array"){
					^for[i](0;$list-1){
						$node[$list.$i]

						$result.[$i][^_xml2hash[$node]]
					}
				}
				
				^case[DEFAULT]{
					^for[i](0;$list-1){
						$node[$list.$i]

						$result.[$node.nodeName][^_xml2hash[$node]]
					}
				}
			}
			
			^if(!$list){
				$result[^xNode.selectString[string(.)]]
			}
		}

		^case(3){
			$result[$xNode.nodeValue]
		}

		^case(9){
			$result[^_xml2hash[$xNode.documentElement]]
		}

		^case[DEFAULT]{
			^throw[...;...]
		}
	}
	
	^Rusage:compact[]
#end @_xml2hash[]

Загружается 1 файл

Так, я сам нашёл пароль:-)

1. парсер пришёл в начало @main[] -- mu: 164.992
2. загрузил xml с диска в память (как объект) -- mu: 836.508
3. для каждого узла /statuses/status (28 итераций) выполнил @_xml2hash -- mu: 6730.11

http://api.twitter.com/1/statuses/user_timeline.xml?count=200&user_id=12 - без регистрации в API и на сайте

Мало итераций

Т.е. вы предлагаете XSLT трансформацией в некий tab-delimited формат а потом парсером? Быстрее будет?

196 итераций и mu: 194306 (сильно!)

Именно

tab-delimited можно и сразу в базу через LOAD DATA

думаю, что тут проблема в xpath

аналогично

@USE
Erusage.p
Node.p


@main[]
$x[^xdoc::load[x.xml]]
^memory:compact[]
^Erusage:measure{
	$h[^_xml2hash[$x]]
}[v]
#^dstop[$h]

Time: $v.time ms<br />
BL: $v.memory_block<br />
KB: $v.memory_kb<br />
ever_allocated_since_compact: $status:memory.ever_allocated_since_compact<br />

@_xml2hash2[xNode][i;result;node]
	^switch($xNode.nodeType){
		^case(1){
			$result[^hash::create[]]
			$i(0)
			^if($xNode.nodeName eq "ids" || ^xNode.getAttribute[type] eq "array"){
				^Node:foreachChild[$xNode;;node;]{
					$result.[$i][^_xml2hash2[$node]]
					$i($i+1)
				}
			}{
				^Node:foreachChild[$xNode;;node;]{
					$result.[$node.nodeName][^_xml2hash2[$node]]
					$i($i+1)
				}
			}
			^if(!$i){
				$result[$xNode.firstChild.nodeValue]
			}
		}

		^case(3){
			$result[$xNode.nodeValue]
		}

		^case(9){
			$result[^_xml2hash2[$xNode.documentElement]]
		}

		^case[DEFAULT]{
			^throw[...;...]
		}
	}
#end @_xml2hash2[]

$.0[
		$.created_at[Wed Jun 16 15:04:36 +0000 2010]
		$.id[16312661011]
		...
		$.in_reply_to_screen_name[void]
		$.contributors_enabled[
			$.id[12]
			$.name[Jack Dorsey]

$.0[
		$.created_at[Wed Jun 16 15:04:36 +0000 2010]
		$.id[16312661011]
		...
		$.in_reply_to_screen_name[void]
		$.user[
			$.id[12]
			$.name[Jack Dorsey]

_xml2hash:
Time: 0.579 s
MU: 187508 (локальный result)

xslt (прямыми руками*) + process + всякие хитрости:
Time: TT: 0.063 s
MU: 22245

локальный result сразу выкидывает все whitespace

-	^if($xNode.nodeName eq "ids" || ^xNode.getAttribute[type] eq "array"){
+	^if($xNode.nodeName eq "ids" || ($xNode.attributes && ^xNode.getAttribute[type] eq "array")){

-	^if($xNode.nodeName eq "ids" || ^xNode.getAttribute[type] eq "array"){
+	^if($xNode.nodeName eq "ids" || $xNode.attributes.type.nodeValue eq "array"){

process быстрее будет чем чтение tab-delimited?

В моем случае LOAD DATA не поможет, т.к. много операций по подготовке данных и без них не обойтись :(

не уверен

с tab-delimited у вас тоже будут небольшие проблемы...

Но в любом случае xslt будет быстрее и по памяти нормально