Monthly Archives: November, 2007

Procedure for amnesia recovery

Written by Marcelo Leal on November 19, 2007 — 1 Comment

I’m deploying a two-node Sun Cluster environment, and while making some tests in such env, i got into interesting situations. One of them was the “amnesia” problem, and i post this in opensolaris forums. Thanks to Ira Pramanick that wrote this procedure to recover from such a scenario: ” Hi, Running into a scenario where [...]

Posted in General, Opensolaris, Suncluster

Sean: I read the Solaris performance tuning book (Adrain Cockscroft) over 10 years ago, way before ZFS. A few clarifications – swapping is bad, moderate paging is not necessarily bad. Paying a few extra hundres or thousands bucks for a few GB RAM is OK, paying a few hundred thousands for a few hundred GB RAM is not. Yes, there is always a trade off. Anyway, I don’t mean anything hostile, just would like you remind you the side effects of tuning down the queue depth.
Sean: In other words, all your servers have been oversized. I am pretty sure the sales folks were pretty happy about it :-) BTW, paging and swapping are two different concepts, I am pretty sure the solaris internals book or solaris performance tuning books cover the topics.
Sean: When I mentioned paging/swaping, I meant “in the case when paging/swaping happens” – nobody wants this to happen but it does happen. In this case, it does require a higher throughput, and a short queue depth will make the performance VERY bad. See Technical Instruction Note: 1011936.1 for an example.
Sean: I suppose any tuning will have impact on other areas of the system. When you tune down the i/o queue, it may help on latency, but I would speculate it will negatively impact throughput. Further more, I remember there was an issue with swap/paging performance when you tune down sd queue depth. So if your swap is on zvol, and you tune down zfs_vdev_max_pending, and the system starts paging, I am pretty sure the performance will be bad.
Robert: Very interesting. When reading your post and interpreting the results (probably wrong) it reads like “ZFS tries it’s best to maintain data integrity by doing L0 + L2/ L1 + L3 updates and the sd driver resorders transactions for the sake of throughput”. Is this correct ? Does Opensolaris have no concept of barriers (reorder domains) as Linux has ?
Carlos Mafort: Marcelo, Comprei seu livro no Submarino e estou aguardando a entrega. Quero conhecer um pouco mais a fundo o ZFS, pois estou querendo montar um NAS (FreeNAS ou Nexenta-baseado no OpenSolaris) e estou em dúvida sobre qual dos dois é mais estável… tem alguma dica? O port do ZFS para o FreeBSD é confiável?
André Alves: Olá… lí uma nota do seu livro na revista Locaweb ed. 22. Sou um entusiasta de novas tecnologias e assim que conhecí o ZFS fiquei impressionado. Seu livro já é uma compra certa. Espero poder trocar informações futuramente. Até mais.
Patrick T.: Great work, I followed all of your work on ZFS since last year, and let me say how much i appreciated your work. As many others, i would probably purchase your book as soon as an english copy will be printed. Really thank you from the University of Verona Patrick T.
M: Nao gosto de comédias brasileiras, aqui vai meu top five: 1º O Pagador de Promessas 2º Cidade de Deus 3º Assalto ao trem Pagador 4º O Bandido da Luz Vermelha 5º Beijo no asfalto
Thomas Widhalm: I can imagine losing one out of many (or two) L2ARC devices would result in just partial performance loss. But is this a verified fact? I think there is a chance that removing one device out of an array, especiall when striped, could force ZFS to rewarm the L2ARC. What has the biggest impact by the situation you stated would be the bigger S7000 Unified Storage Systems. The SSDs containing the L2ARC are built into the heads of the storage. So a failover of a pool will remove all SSDs from it. By now I have not found a way to provide a pool with any sort of L2ARC when failovered.
uep: +1 Agree fully. However, your note does also recognise the fact that you’re talking about a warmed l2arc. The same problem will hit the application at startup when the arc is cold (until persistent l2arc lands). Also, if you stripe across multiple l2arc devices, you spread load across them but you also spread failure impact over them. If one of four l2arc devices disappears, you’ll only have a cache miss rate / performance impact 1/4 of the drop you’re concerned about, until steady state is reached again. Thus, 4 cheaper l2arc devices with moderate IOPS may really be better than 1 or 2 faster ones.
Constantin Gonzalez: Hi Marcelo, great post, and a sharp analysis! I agree that L2ARCs should be made mirrorable to avoid sharp declines in performance upon loss. OTOH, what happens if you use 2 L2ARC devices, non-mirrored? Statistically speaking, cache data will end up on both devices, equally distributed. Now if you lose on of them, the performance degradation will be 50% of the case of when you only had 1 device. 33% for 3 L2ARCs and so on. Given that read-biased SSDs are cheaper than write biased ones (you can get away with MLC, write performance isn’t important etc.), this may be an acceptable alternative. In extreme cases, for large servers, you could have 10 L2ARCs and then you’d hardly notice the 10% performance loss when one of them goes out. Cheers, Constantin
Software: Congratulations! You have just won a new feed reader :) .. really delicious blog, Mike.
Robert: Nice trace. I wonder why the disks were spinning at all – even with the problem in place. Shouln’d the ZFS ARC and L2ARC handle the (oviously useless) repeading calls to get the attributes ? Shouln’t the attributes be cached on the storage side ?
renata araujo: Adorei seu blog, deixei nos meus favoritos! parabéns, quando der visite o meu blog, bjs..
Orphen D: Poderia me repassar seu email quero trocar ideias de linux quero aprender como editar código do linux essas coisas meu msn axanka2@hotmail.com
sid wilroy: I have been enjoying the Solaris 10 ZFS Essentials https://memberservices.informi t.com/buy.aspx?isbn=0137000103 &eb=cb574f95-425e-4f47-8f3 8-c9201a902eda&partner=53& amp;cmd=add
Blake Butler: Some people think that car insurance is a waste of money but it is really very essential that you have one.-”‘
Sri: Cleanely done and written. Very nice. Any reason behind not using opensolaris?
Krissy Vickerman: @jimm i see now you are right because this vid proves the point made http://www.youtube.com/watch?v =1NVuUAr4MMs
David: Thanks – very helpful.
Aşk Büyüsü: im having the same problem using opensolaris 2009.6…. any solutions??
Eric Sproul: Hi Marcelo, I’ve downloaded the PDF and look forward to reading it. We use ZFS extensively here and like you I am waiting to see what Oracle wants to do with OpenSolaris. Cheers, Eric
Jaquelyn Oesterle: I like your blog very much, thanks you
Eduardo: Este livro é destinado ao ensino de administração do sistema opensolaris?
Filipe Brandenburger: Ae Velhinho! Parabéns pela publicação do livro! Muito legal. Agora é só disponibilizá-lo sob Creative Commons… hehehehehe. Abração! Filipe
Ryan Depaulis: This is awesome , I’m sure that technology will get much much better.
Gama: Excelente! Na minha próxima atualização de SO, meus HDs rodarão em ZFS. Obrigado!
Mike: Thanks for a wonderful post, l ve been looking for such information, I will join jour rss feed now.
Chris: apparently i can update the contents of the cluster database via dscfg -C – -a file however this appears to be only updating dscfg_local arggh :( time to give up…
Chris: well i tried using the supported stuff from sun website and i have the same problem… what i have noticed is that prior to running sndradm if i run dscfg -l -s /dev/did/rdsk/d5s0 | grep -v “#” – the contents is empty. Once i setup the replication with sndradm and rerun the dscfg command it returns one line with setid: 1 setid-ctag i would of thought it would return the same as /etc/dscfg_local which in my case has: cm: 128 64 – – – – – – - sndr: nas1 /dev/rdsk/c1t1d0s0 /dev/rdsk/c1t1d0s1 nas2 /dev/rdsk/c1t1d0s0 /dev/rdsk/c1t1d0s1 ip sync tank1 – setid=1; - sv: /dev/rdsk/c1t1d0s0 – - sv: /dev/rdsk/c1t1d0s1 – - dsvol: /dev/rdsk/c1t1d0s0 – sndr dsvol: /dev/rdsk/c1t1d0s1 – sndr so perhaps the problem isnt that configuration is lost, but that configuration isnt even being written to the cluster database on the shared disk??? wonder how i could manually get that info on the shared storage.. i...
Chris: from all the googling i did find alot of emails from you and alot of pushback from devs saying it wasnt a problem, however here i am with the same problem as you, using opensolaris 2009.6.. :( i may give this a shot with the latest solaris release and the packages from the sun website, see if its just an opensolaris issue. such a shame that this problem is here as your solution is fantastic providing AVS works – which it does until a node restarts!!! i gather your current agent code is here “http://www.eall.com.br/ /hp/Solaris/MRSLnonshareddevic e-2.2.tar.gz” ? once again… awesome work with the agent!
Knut: I don’t get, why Schumacher had a bad idea. If it don’t works from the first try, why should one give up ? If you don’t get an improvement after you tried your bag of tricks, it’s time to think about another job/hobby. BTW: I don’t think, Schumacher tries to impress us. He likes to drive fast and agile formula 1 cars. And as an employee of a f1 team, he even gets paid for the thrill. That doesn’t sound like a bad idea to me.
Chris: im having the same problem using opensolaris 2009.6…. any solutions??
Chris: Hi, this is a fantastic agent, and i have it up and running so far flawlessly however when my cluster restarts AVS appears to loose its configuration? if i run “sndradm -C local -g POOLNAME -n -u” im told: Remote Mirror: avs1 /dev/rdsk/c8t1d0s0 /dev/rdsk/c8t1d0s1 avs2 /dev/rdsk/c8t1d0s0 /dev/rdsk/c8t1d0s1 sndradm: warning: SNDR: /dev/rdsk/c8t1d0s0 ==> /dev/rdsk/c8t1d0s0 not already enabled dsstat also returns nothing… so far my solution is running on both nodes: dscfgadm -d rm /etc/dscfg_cluster && rm /etc/dscfg_local echo “/dev/did/rdsk/d2s0̸ 3; > /etc/dscfg_cluster dscfgadm and then syncing the primary to the slave again.. obviously not ideal :( any suggestions?? and cheers for this awesome agent… would be nice if the AVS comunity got behind this!!
Hosam Al Ali: Nice one and thanks for this Job best,
Roman: As Shumi/Ferrari fan I can say that Alonso appear in Ferrari is very strange. They should put all efforts to Massa. Go Massa, Go!
Damian Wojsław: Now I need to learn Brazilian portugese. One more on the list.
benr: Nice! You’re the man Marcelo! I hope I can get an autographed copy some day. :)
Jim Grisanzio: This is excellent news, Marcelo. Congrats!
Igor: It would be great if English translation was printed – as it seems there’s no book about this subject in English and most people can’t read Brazilian Portuguese. And I can assure you’ll sell many more copies in English.
Dave Miner: Congratulations, Marcelo! Though I can’t read the book, I know how much work such a project can be, and how satisfying it is to see it in print. I hope it sells well, as ZFS deserves a lot more than the space I was able to give it in OpenSolaris Bible.
Leonardo: Olá Marcelo Leal … Sou muito inexperiente no Linux, acabei de instalar o Ubuntu 9.10 e como todo mundo que começa neste sistema, “estou com problemas com minha rede sem fio” Tipo, quando inicio o pc, ele reconhece que tem rede, mias não consegue conecta-se, queria saber como faço para configurar ela ? há, gostei da sua participação no olhar digital, valeu, abraços, aguardo respostas …
Janessa Edelbrock: [..] A little unrelated, but I absolutely liked this blog post [..]
Marcelo R: Olá Marcelo tudo bem? tenho certeza que sim! me tira uma dúvida que mesmo procurando em fórum e afins não consegui entender. Tenho o Windows 7 como sistema operacional padrão, porém ontem particionei minha HD em 3 unidades no caso C: ; D: e M: agora gostaria de instalar o Ubuntu na partição M: que no caso deixei com 10Gb para isto, quando dou o boot com CD para instalação seleciono para gravar na HD porém ele não me dá opção das que particionei, para no caso estar instalando o mesmo na M: como disse, agora vem, como faço isso ???? desde já agradeço a atenção.
Ana: oi , eu fiz uma e agora não consigo acessar. você sabe o que aconteceu? você também fez?
Nildo oliveira Says: Bom dia Marcelo obrigado desde ja pela atenção a resolução esta em 800×600 falaram que o 8.04 funciona bem em relação esse probleminha vou tentar essa do Driver vesa
nildo oliveira: Ola Marcelo vi seus videos no Olhar Digital bacana sua participação baixei a ultima versão Ubuntu 9.10 Karmic Koala mais estou tendo um problema com a resolução da tela,entrei no Ubuntu br /forum e vi que é um problema comum,como só usei vagamente a Kurumin não sei mudar resolução e o Ubuntu não reconhece meu munitor. a ação xorg.conf não funciona um usuario disse para eu digitar: lspci | grep “VGA” e ai apareceu isso no terminal: 00:02.0 VGA compatible controller: Intel Corporation 82G33/G31 Express Integrated Graphics Controller (rev 10) por favor me ajude marcelo. obrigado desde ja e desculpa te aluguar assim se quiser responder por email….nildoliveira@ymai l.com
phil: will be interested in more details of p33/opensolaris/ps3MS. have you looked into minidlna? http://minidlna.sourceforge.ne t/ seems small and simple.
Marco: We used ZFS for over 3 years facing many problems and spending a lot of time. Then we went with Netapp and saved time and money..