This is how you can override the Edit action for Book__c custom object. If the record type is Alpha you will be directed to a new page, otherwise you go to the standard edit page.
<apex:page standardController="DaizyLogik__Book__c" action="{!URLFOR(
CASE(DaizyLogik__Book__c.RecordType.Name, 'Alpha', $Page.BookAlpha,
'/' + DaizyLogik__Book__c.Id + '/e?nooverride=1'),
DaizyLogik__Book__c.Id) }">
<apex:variable value=”{!DaizyLogik__Book__c.RecordType.Name}” var=”recTypeName”/>
</apex:page>
by Vladimir Martinov
DaizyLogik 