Puzzle this out

So did we ever finish the puzzle? Well 'yes' and 'no'.
We put all the pieces in. All that we had that is, and all that had a place to go.
We were left with 1 spare piece and 4 holes. The spare piece did not fit in any of them.

Puzzle plus 1 minus 4 (the arrows are my later addition...)

In fact I found where the spare piece went but we had to remove a piece to put in in, the two were identical.
Someone has 4 of our pieces and we have one of theirs!