Others say those buildings should be restored and brought back to life in recognition of St. Joseph's rich history.
With downtown St. Joseph's Gilmore Building as an example, the city's preservationists are at work.
The look inside the building at 107 S. 6th St. looks dramatically different than it did less than one year ago.
"It's night and day," said Beau Walker, a new tenant in the renovated building. "From what it was to what it is today, it's totally different."
The new Paradox Theater and Coffee House located on the first floor is part of the building's extreme makeover.
"(The building) was taking on water," said Scott Des Planques, the city's historic preservation planner. "The roof leaked, some of the floors needed to be fixed. Windows were literally falling out of the panes into the street below and had to be boarded up."
The Gilmore Building is one success story for St. Joseph's Endangered Building Program. After receiving the endangered designation, the building's owner became eligible for Save Our Heritage grant money and collected $45,000 to help with renovations.
"We try to identify problem properties in hopes they will be worked on and brought back to their historic standards."
Any building more than 50 years of age is eligible for the designation. The goal is to save a link to the city's past.
"We keep losing them a little bit at a time," Des Planques said. "We'd like to turn that around and create heritage tourism where people want to come to St. Joseph and see block after block of historic buildings as they were."
"We're trying to look for a neighborhood look that helps it lock into the historic correctness of that era that it was built so that when visitors come, they can see the streetscape like it was," said Ellis Cross, chair of the city's Landmark Commission.
There's no shortage of homes in St. Joseph that would qualify.
"There is such a concentration of historic buildings here between 28th Street and the river," Des Planques said. "I've never seen such concentration still standing."
"They over-engineered everything," Cross said. "These homes that have been there since the 1850s to 1880s are solid as the day they were built. You're not finding that in brand new stuff."
For the Gilmore Building, which also dates back to the 1800s, it's endangered no more. Now it's goal is to bring more people downtown.
"(The building's renovation) sparks interest for a lot of reasons. I see us contributing to that conversation in that way," Walker said.
The Paradox Theater and Coffee House shows art and hosts music and other performance events.
The building's owner is in the process of renovating the upper levels of the building, turning the space into apartments.
Nominations for this year's Endangered Buildings awards need to be turned in to the Historic Preservation office at city hall by April 11.