I've been in Commercial Real estate for 15 years and own my own brokerage in SoCal. My wife owns her own Financial Planning company as well.
We both feel that having a home paid for is a good thing. There are many arguments as to why you shouldn't, but there is a tremendous piece of mind that comes with having your home free and clear.
However, here are some reasons why you should't pay your house off.
My wealthy friends all have fairly significant debt on their homes. Primarily because those who may want to sue you (this day may come), the first thing they do is look to see how much equity you have in your home. Those with debt are probably less likely to get sued.
One thing I would seriously consider...and this is totally up to you and your pallet for investing and risk. I would consider getting a mortgage on this house and buy an apartment building with your cash. You can easily buy a $600k apartment building and put 40% down while keeping some cash in the bank.
In California a decent return on an apartment building is about 5% of your down payment...but you also pay litteraly no taxes on those returns due to a deduction called "depreciation." $250k down with 5% returns is $12,500 a year. In other states returns on apartments is well over 7 and 8%. The apartment market (5+ units) is down right now. You can get a really nice return with minimal risk and have a chance at some significant gains in value over the next 5-10 years.
Real estate is a thirsty animal though. It's also a bit of a bitch. But if there ever were a time to buy a nice building now is that time.
Or buy a 10-15 unit apartment building and live in a unit. I'm not into self managing real estate but it would be a really solid learning experience for you.