Your clutch fork could be bent/broken as well. Starting it in neutral or shifting TO neutral should be fine. But getting it into any other gear should be problematic.

Shifting while you're moving, you're probably near-rev-matching. Try revving it higher and then shifting, you'll probably get some grinding.